acceptodds
Under review as a conference paper at ICLR 2027

Attractor Degeneracy: A Theoretical Analysis of Equivalent Memory Classes in Hopfield Neural Networks

Abstract

Hopfield neural networks (HNNs) realize associative memory through dynamics induced by memories. Different memory matrices can nevertheless generate conjugate network dynamics, a phenomenon we call attractor degeneracy. We study the equivalence relation generated by row and column permutations and sign flips in binary HNNs. With synchronous updates and spin retention at zero field, equivalent matrices have update maps related by a signed coordinate relabeling. This bijection preserves trajectories, periodic attractors, basin sizes, and Hamming distances. We describe the equivalence classes as group orbits, derive an exact count among all binary memory matrices with memories and neurons, and give an orbit-recognition method that recovers the row and column transformations. The count avoids enumerating the memory space, while a recovered transformation allows the dynamics of one representative to be transferred to any equivalent memory matrix. Exact complete-space and independently sampled candidate-bank experiments preserve the reported statistics while reducing repeated HNN evaluations beyond a column-only quotient.

Primary area:
learning theory
Keywords:
Hopfield Neural Networks, Equivalent Memory Classes, Degeneracy, Group Orbit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K8cCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the symmetries of classic Hopfield Networks corresponding to signed permutations of neurons and stored memories. It develops some group theoretic characterizations of these symmetries, including a mechanism for "normalizing" sign flip operations so that they can be safely removed from subsequent analysis. This leads to an algorithm for checking if two Hopfield Network weight matrices are equivalent under these symmetries, based on solving a colored bipartite graph isomorphism problem. The algorithm is still worse than polynomial time because it enumerates row permutations, but is shown empirically to be orders of magnitude faster than a less sophisticated brute force check.

Critical strengths
  • For readers unfamiliar with common neural network symmetries, this paper is a potentially useful and current characterization.

  • As far as I could tell, the mathematics in this paper appears to be technically sound, although some is rather complex and difficult to check.

  • The sign flip normalization trick is potentially novel, and the contributed algorithm exhibits a clear empirical improvement in run time over a brute-force approach.

  • The submission includes some of the code needed reproduce the results, which does not require GPUs or any other prohibitive computational resources.

Critical weaknesses
  • The motivation and significance of this work is not clear. It would make the paper more compelling if it explained a concrete application in which multiple Hopfield weight matrices are present and one needs to determine their symmetry equivalence relationships.

  • I believe the symmetries discussed in this paper, and several of the derived theorems, are already well-studied and common knowledge to researchers in this area. For example, a quick search turned up the paper below that was not cited as related work. Combined with the fact that the contributed algorithm is still worse than polynomial time, it is unclear whether this paper makes a significant new contribution.

Dong J, Xu S, Chen Z, Wu B. On permutation symmetries of Hopfield model neural network. Discrete Dynamics in Nature and Society. 2001;6(2):129-36.

  • The provided code only includes function definitions for the core algorithms. It does not include the code that calls those functions to regenerate the empirical results.

  • Some of the mathematical jargon seems unnecessarily complicated and possibly misleading. For example, the paper uses "attractor degeneracy" to refer to the equivalent dynamics of two symmetric networks, but this term might be easily confused with issues related to spurious attractors.

Questions for authors
  • Theorem 7 only refers to a "target anchor" but immediately after it says "one source anchor suffices." what is the definition/role of the source anchor? is it (1,1)?

  • Can the authors elaborate on concrete motivations for this work? In what context would one have multiple Hopfield weight matrices, and want to determine if they are symmetric?

  • Can the authors expand on what their work contributes beyond what is already known about neural network symmetries, such as the paper cited above?

4 comments
AuthorsOpenReview ↗

Response to Reviewer K8cC

Dear Reviewer K8cC:

Thank you for your comments on sign normalization and for pointing out the missing experiment scripts. Our study began with an observation: randomly drawn memory matrices could look quite different yet give the same statistics. This led us to look for symmetries in the memories themselves. We show that memory matrices related by these symmetries induce the same dynamics up to a relabeling of states, so evaluating one representative gives the recall score for the entire class.

Responses to Weaknesses

W1. About motivation and significance

When choosing test cases for a comparison of retrieval rules, a researcher may seek memory sets whose stored patterns are stable under the Hebbian rule and whose networks have few spurious fixed points. Each candidate set induces a Hebbian weight matrix. Our method identifies candidates in the same memory orbit, where both properties are preserved. Screening one representative therefore suffices for the whole orbit.

W2. About On permutation symmetries of Hopfield model neural network

We had not seen On permutation symmetries of Hopfield model neural network before submitting our paper. We thank the reviewer for drawing our attention to it. Their result shows that a given permutation of the neurons carries the attractors and basins of one network to those of the permuted network. This gives useful context for the neuron-permutation case of our dynamical result. If our paper is accepted, we will cite Dong et al. in the final version and explain this connection.

The complexity question matters because recognizing an equivalent candidate must cost less than evaluating it again to save time. The general procedure examines at most MNMN target anchors, with graph search that may branch at each one. It returns an exact answer and, on a match, a signed row and column transformation that can be checked entry by entry (Appendix B).

W3. About experimental code

You are right about the missing scripts. We included the core routines and overlooked the scripts that run Section 6. If the paper is accepted, we will make the experiment code public, with the scripts, settings, and commands needed to reproduce Figure 3 and the corresponding tables in Appendix C.

W4. About "attractor degeneracy"

We see how “attractor degeneracy” could suggest a spurious attractor. In Section 1 the term refers to the same attractor structure appearing in networks built from different memory matrices. A spurious attractor is judged against the memories stored within a single network. Definition 1 gives the precise equivalence relation used in the paper.

AuthorsOpenReview ↗

Response to Reviewer K8cC

Responses to Questions

Q1. what is the definition/role of the source anchor? is it (1,1)?

Yes, you have identified the source anchor correctly. We fix (1,1)(1,1) in Ξ1\Xi_1 and use its row and column for sign normalization, making all entries in both positive. Their positions in Ξ2\Xi_2 are unknown. In Eq. (21), pp and qq map target indices to source indices. The target anchor satisfies p(a)=q(b)=1p(a)=q(b)=1, so (a,b)=(p−1(1),q−1(1))(a,b)=(p^{-1}(1),q^{-1}(1)).

Appendix B.4 gives p(2)=q(4)=1p(2)=q(4)=1, so the fixed source anchor (1,1)(1,1) corresponds to target anchor (2,4)(2,4). Searching the target positions covers every possible correspondence while keeping one source anchor.

Q2. Can the authors elaborate on concrete motivations for this work? In what context would one have multiple Hopfield weight matrices, and want to determine if they are symmetric?

In our experiments, randomly sampled memory matrices sometimes looked quite different yet gave the same statistics. This led us to ask which changes to the stored patterns preserve the underlying dynamics. That question also arises when choosing memories for a retrieval study. Dong et al. (2001, Section 4), for example, construct a memory set whose patterns are stable and have attraction basins of equal size.

Consider selecting test cases for a comparison of retrieval kernels. One may want a reference set for which the classical Hebbian network stores every intended pattern as a fixed point and has few additional fixed points. Here we count a fixed point as spurious if it is neither a stored pattern nor the negative of one. Each candidate set generates its own weight matrix, and deciding whether to keep it requires checking these properties.

Our classification makes that decision once per memory orbit. The state relabeling maps fixed points bijective, and it also maps the stored patterns together with their negatives onto the corresponding set for the other candidate. The two candidates therefore agree on whether every stored pattern is stable and on how many spurious fixed points remain. Once a representative has been screened, every candidate in its orbit inherits the same decision. A researcher can then select representatives from the qualifying orbits for the subsequent kernel comparison.

Q3. Can the authors expand on what their work contributes beyond what is already known about neural network symmetries, such as the paper cited above?

One theoretical contribution is to describe memory equivalence through a group action and show which dynamical properties are shared by the networks in each orbit. Dong et al. show how a given neuron permutation relates two networks and use symmetry to construct stored patterns with equal basin sizes. The repeated statistics in our experiments led us to ask which memory matrices are related by changes to the stored patterns or neuron coordinates, and what those networks share throughout their dynamics.

Harrison’s work on counting classes of binary matrices suggested formulating this question as a group action on the memory space. Starting from one matrix, we can reorder memories, reverse the sign of a memory, permute neurons, or reverse one neuron’s sign across all memories. All matrices reachable by composing these operations form its orbit. This viewpoint lets us ask which dynamical properties can be determined from a single representative of each orbit. In the relation Ξ2=PΞ1Q\Xi_2=P\Xi_1Q, the row operations in PP leave the Hebbian weights unchanged, and the column operations in QQ relabel both the weights and the states.

The dynamical link is exact for the update rule studied here. Let FΞF_{\Xi} denote the synchronous Hebbian update with spin retention at zero field. For any two memory matrices related as above, Theorems 1 and 6 give FΞ2(VQ)=FΞ1(V)QF_{\Xi_2}(VQ)=F_{\Xi_1}(V)Q for every state VV. Thus two corresponding initial states follow corresponding trajectories at every update. The same map pairs the attractors and their basins. This is a stronger statement than agreement of the statistics we first observed.

In an exhaustive study, Theorem 5 tells us exactly how many representatives are needed under these four transformations. In a random search, Theorem 7 lets us decide whether a new memory matrix belongs to a class we have already tested and, on a match, recover a transformation that can be checked entry by entry. Lemma 7 explains why the earlier score can be used: each corrupted cue returns to its designated target if and only if the corresponding cue does.

I appreciate the authors' response, but I am planning to leave my score unchanged for the following reasons:

  • The authors state they will make some changes, but it appears they did not actually post a revision to the manuscript
  • The main motivation they provide is "comparing retrieval rules." However, depending on the structure of a retrieval rule, its properties may depend on the weight matrix itself in a way that is not invariant across the dynamical symmetry equivalence class. Furthermore, the super-polynomial time check still makes it difficult to apply the method in practice for this kind of comparison.
  • I still feel that the types of symmetries studied (signed row/column permutations) are common knowledge to the ICLR community and this paper will not provide that community with much new information.
AuthorsOpenReview ↗

Response to Reviewer K8cC

Dear Reviewer K8cC:

Thank you for following up. We have uploaded the revised manuscript. Section 2 now cites Dong et al. (2001) and briefly explains how their result on neuron permutations relates to ours. We have also added their paper to the reference list. Thank you for bringing this work to our attention.

1.

In our example, we first select memory sets with a desired property under the Hebbian rule and then use those sets to compare retrieval rules. Our result makes the selection easier. One representative per memory orbit suffices to determine which candidates pass the Hebbian screening. The selected cases can then be tested under a different retrieval rule. Reusing that rule’s scores would require checking that its update and scoring procedure respect the same symmetry.

Consider these two memory matrices, each storing five patterns on four neurons:

Ξ1=(−1−1−1−111−1−11−11−11−1−111111),Ξ2=(−1−1−1111−111−1111−1−1−11111).\Xi_1=\begin{pmatrix}-1&-1&-1&-1\\\\1&1&-1&-1\\\\1&-1&1&-1\\\\1&-1&-1&1\\\\1&1&1&1\end{pmatrix},\qquad \Xi_2=\begin{pmatrix}-1&-1&-1&1\\\\1&1&-1&1\\\\1&-1&1&1\\\\1&-1&-1&-1\\\\1&1&1&1\end{pmatrix}.

In each matrix the first four rows contribute 4I44I_4 to the Gram matrix, and the fifth row is the same. Setting the diagonal to zero gives the same Hebbian weight matrix in both cases:

WΞ1=WΞ2=(0111101111011110).W_{\Xi_1}=W_{\Xi_2}=\begin{pmatrix}0&1&1&1\\\\1&0&1&1\\\\1&1&0&1\\\\1&1&1&0\end{pmatrix}.

From any given initial state, the two networks follow exactly the same trajectory. To see what the choice of memories still changes, we enumerated all 16 initial states. For each state VV, we computed its minimum Hamming distance to a stored row ξa\xi_a or its negative:

d_{\pm}(V,\Xi)=\min_{1\le a\le 5}\min\bigl\\{d_H(V,\xi_a),\\,d_H(V,-\xi_a)\bigr\\}.

Minimum distanceNumber of states for Ξ1\Xi_1Number of states for Ξ2\Xi_2
0810
186
200

The first and fifth rows of Ξ1\Xi_1 are negatives of each other, so its five rows represent four distinct pairs of the form ξ,−ξ\\{\xi,-\xi\\}. The five rows of Ξ2\Xi_2 represent five such pairs. This accounts for the eight versus ten states at distance zero. Signed row operations preserve the set of pairs, and signed column operations preserve Hamming distance under the corresponding relabeling of states. The distance histogram is therefore constant within a memory orbit. The two histograms distinguish two memory orbits with exactly the same WW.

This example explains why we keep track of the memories that generated WW. Within a memory orbit, Lemma 7 pairs each stored target and its perturbed cues with their counterparts. Whether a cue returns to its designated target is then the same in both networks, which is precisely the outcome we reuse.

2.

Direct enumeration has 2M+NM!N!2^{M+N}M!N! signed row and column transformations to try. Our general procedure can reject a pair before graph search if its row or column correlation multisets differ in absolute value. The candidate-bank experiment uses a simpler exact test. At a fixed memory count MM, the number of column types up to sign and the possible signed row actions depend only on MM. Computing the orbit key therefore takes O(N)O(N) time per matrix as the neuron count NN grows. For M=3M=3, we scan the columns and sort just four counts. Two candidates are in the same orbit exactly when those sorted counts agree. At N=64N=64, the median end-to-end speedup over signed-column caching was 4.484.48 times across ten banks of 1,000 independently sampled candidates, including the cost of computing the keys.

3.

Even when a conclusion seems obvious, intuition can be wrong. Our proof shows when these symmetries preserve recall outcomes. Once we know the conditions, we can reuse recall results in networks that meet them and know what to check when we change the update rule. The group action gives an exact orbit count, and the recognition algorithm tells us when a new candidate belongs to an orbit we have already evaluated.

Reviewer_GtqzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies equivalence classes of memory matrices in classical binary Hopfield neural networks. It proves that signed row and column permutations induce conjugate dynamics, preserving trajectories, attractors, and basin properties. Using group actions and Burnside's lemma, the authors derive an exact orbit-counting formula and develop a recognition algorithm to reuse computations across equivalent memory matrices. Experiments demonstrate reductions in redundant dynamical evaluations.

Critical strengths
  1. Clear theoretical formulation: The paper provides a systematic characterization of memory equivalence using group actions, with formal results on dynamical conjugacy and orbit counting.

  2. Computational reuse: The proposed orbit-recognition method enables redundant dynamical evaluations to be avoided, with empirical evidence of computational savings.

Critical weaknesses
  1. Limited theoretical novelty: The dynamical equivalence under signed permutations follows naturally from the algebraic symmetries of the classical Hebbian weight matrix. Although the exact orbit-counting formula is useful, the paper does not sufficiently establish how its theoretical insights extend beyond known symmetry and group-action principles.

  2. Limited scope and broader significance: The analysis is restricted to binary Hopfield networks with Hebbian weights and a specific synchronous update rule. The experiments primarily demonstrate computational reuse within this setting, without establishing broader implications for associative memory capacity, learning, or retrieval performance.

Questions for authors
  1. What is the main theoretical insight beyond applying established symmetry and group-action techniques to the classical Hopfield model? In particular, which results are specific to Hopfield dynamics rather than general consequences of matrix symmetries?

  2. Can the authors clarify whether the proposed equivalence classification or computational reuse extends to more general associative memory models, such as modern Hopfield networks?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Gtqz

Dear Reviewer Gtqz:

Thank you for your comments on the dynamical result and its use in recall experiments. When we sampled memory matrices, we found that different matrices sometimes gave the same recall statistics. This led us to study which changes to the stored memories preserve the network dynamics and allow a search to reuse an earlier result.

Responses to Weaknesses

W1. Novelty

Our main theoretical contribution is a group-theoretic analysis of the symmetries that arise when stored memories or neuron coordinates are reordered or sign-flipped in a binary Hebbian Hopfield network. A feature of these operations matters for the classification. Flipping every row and then every column leaves the matrix unchanged, since each entry changes sign twice. We account for this redundancy and show that among all matrices related by sign flips alone, exactly one has only positive entries in its first row and first column. This lets us handle sign flips first and then study how permutations act on the normalized matrices. From that action we obtain the exact number of classes for any numbers of memories and neurons (Section 4). For two given matrices, we can also decide whether they are in the same class and recover the operations relating them (Section 5).

W2. Scope and broader significance

A concrete use is choosing memory sets for a comparison of retrieval rules. A researcher may want test cases in which every stored pattern is a fixed point under the Hebbian rule and there are few spurious fixed points. Here a fixed point is spurious if it is neither a stored pattern nor its negative. Within a memory orbit, the state transformation pairs the fixed points and also pairs the stored patterns together with their negatives. If one candidate passes this screen, every candidate in its orbit passes. A search can therefore move on to a different orbit.

Responses to Questions

Q1. What is the main theoretical insight beyond applying established symmetry and group-action techniques to the classical Hopfield model? Which results are specific to Hopfield dynamics rather than general consequences of matrix symmetries?

We identify a group action on stored memories and prove that each orbit can be treated as a single class when evaluating recall in the binary Hebbian network. This gives the memory-matrix partition a dynamical meaning and allows us to compute the retrieval score in Section 6 on one representative of each class.

The preservation of Hamming distance in Theorem 2 follows directly from signed coordinate symmetry: applying the same signed permutation to two states preserves the number of coordinates on which they differ. The orbit count in Theorem 5 and the recognition criterion in Theorem 7 are group-theoretic results for the particular action on memory matrices studied here. We derive the count for this action and prove that the anchored graph test recognizes exactly its orbits.

The result specific to Hopfield dynamics is that the coordinate map carries every network update to the corresponding update in the other network (Theorems 1 and 6). For Ξ′=PΞQ\Xi'=P\Xi Q, we prove FΞ′(VQ)=FΞ(V)QF_{\Xi'}(VQ)=F_\Xi(V)Q using the Hebbian weights and the synchronous update rule, including spin retention at zero field. Iterating this identity gives the correspondence of trajectories, attractors, and basins. For retrieval, Lemma 7 also accounts for sign changes of individual stored memories. A target and its cue can be reversed together because FΞ(−V)=−FΞ(V)F_\Xi(-V)=-F_\Xi(V). Consequently, the number of stored rows that are fixed points and the recall score in Section 6 are constant within each orbit.

Q2. Can the authors clarify whether the proposed equivalence classification or computational reuse extends to more general associative memory models, such as modern Hopfield networks?

This paper studies binary Hopfield networks. The group-theoretic approach does extend to modern Hopfield networks. Row and column permutations give a direct example. For a real memory matrix Ξ∈RM×N\Xi\in\mathbb{R}^{M\times N}, permuting rows reorders the stored memories, and permuting columns reorders their coordinates. In the standard modern Hopfield update, recall is a weighted sum of the memories, with weights determined by their dot products with the query. Reordering the rows therefore leaves the result unchanged. If we reorder the coordinates of both the memories and the query in the same way, the dot products stay the same and the retrieved vector follows the same coordinate permutation.

In related work, we use group in a modern Hopfield network to treat different poses of one object as a single memory. For the update that averages over all rotations, rotating the query leaves the probability assigned to each object unchanged and rotates the retrieved state accordingly. Reusing a recall score in a modern model requires checking that its own update and scoring rule preserve the outcome for matching cues and targets.

Reviewer_Vy3tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks at classic binary Hopfield nets (Hebbian weights, synchronous updates, “keep your spin if the field is zero”). It points out that if you shuffle or sign-flip the rows and columns of the memory matrix, you get the same dynamics up to a signed relabeling of neurons. The authors call this “attractor degeneracy.” They count the resulting equivalence classes exactly with Burnside’s lemma, give a graph-isomorphism test that decides whether two memory matrices are equivalent and recovers the transformation, and show that scoring one matrix per class saves compute (M=3 and M=4 experiments).

Critical strengths

Clean and careful. They’re explicit about the zero-field tie rule, and the fact that the recognition procedure returns an actual certificate (P, Q), not just a yes/no, and they verify it entry by entry, is a nice result.

Also, the presentation is good, and the paper reads very well.

Critical weaknesses

Theorem 1 should not be a "Theorem", as it is not a huge results: Hebbian W = ΞᵀΞ ignores signed row permutations, and gets conjugated by signed column permutations. Signed permutations commute with the sign function. It is also highly related to the gauge symmetry spin-glass researchers have been using for a long time.

Counting is not new either: Counting binary matrices under row/column permutations plus complementation has been done in previous work already (Harrison, correctly cited by the authors). Testing equivalence of ±1 matrices via colored bipartite graph isomorphism is the standard trick for Hadamard-matrix equivalence. The novelty is mostly stitching these together: the authors should be more explicit about this.

I may be wrong, but don't the dynamics depend only on W, not on Ξ? Distinct orbits can have the same W, or the same W up to Q, and then their dynamics are identical by construction. The natural quotient is “W up to signed conjugation,” which is coarser and more meaningful. The paper even admits that distinct orbits may still be dynamically equivalent but never chases it, which is something that should be done.

I found the paper nicely written, and quite informative for me, which was refreshing in this round of reviews. However, I'm still a little concerned about the overall "Novelty" of this work.

Minor: Most of the related works section is about modern Hopfield networks, a topic not relevant to the paper.

Questions for authors

Don't the dynamics depend only on W, not on Ξ? (see above for the longer answer)

4 comments
AuthorsOpenReview ↗

Response to Reviewer Vy3t

Dear Reviewer Vy3t:

Thank you for your close reading, particularly your comments on the zero-field rule and the recovered transformation. Our motivation is that a random search over memories can revisit a class already evaluated. The group action describes these classes, and the dynamical proof shows why one evaluation per class suffices. Within each class, the trajectories correspond under a signed relabeling of neurons, and the recall score for the stored targets is preserved. The class count therefore tells us how many representatives suffice to cover the memory space. When a search revisits one of these classes, recognition gives us the transformation that lets the new candidate share the earlier result.

Responses to Weaknesses

W1. Theorem 1 should not be a "Theorem", as it is not a huge results

We agree that the Hebbian weight identity follows directly from the learning rule and that the resulting symmetry is closely related to gauge symmetry. We nevertheless retain the theorem label because the result asserts a precise dynamical statement, not only an identity for WW. Under the four transformations defined in the paper and the specified zero-field rule, for every state VV,

$ F_{\Xi_2}(VQ)=F_{\Xi_1}(V)Q $ whenever $ \Xi_2=P\Xi_1Q. $

This identity establishes a conjugacy of the update maps. Iterating it gives the correspondence of trajectories, and the state-space bijection in Theorem 2 then gives the correspondence of attractors and basins. Those are the dynamical facts that give the counted memory orbits their meaning and justify the later reuse of recall evaluations. We do not present the algebraic identity itself as a new symmetry principle. We use it to establish, for the precise model and update convention studied here, why the memory transformations define classes of dynamically related networks. We therefore consider “Theorem 1” an appropriate label for this central proved result; the label reflects its role in the paper, not a claim that its proof is difficult.

W2. Counting is not new either & Novelty

We first identified the symmetry induced by the specified memory transformations in the Hebbian Hopfield network. In studying how to describe this structure, we found Harrison’s work on binary-matrix equivalence classes informative. Its scope is the combinatorial counting of classes under particular matrix operations. It does not address Hopfield dynamics or the dynamical meaning of memory-matrix equivalence. Harrison’s work inspired our group-theoretic viewpoint, but we did not directly transfer its results to the Hopfield setting. Our work studies the relation between the memory symmetries we identified and the dynamics of the Hopfield network.

W3. Don't the dynamics depend only on W, not on Ξ?

We agree that grouping Hebbian weights up to signed conjugation is natural when comparing state trajectories. Section 4.5 allows distinct memory orbits to induce conjugate dynamics. To reuse a recall score, we need to match the targets and their perturbed cues across candidates. Lemma 7 gives this correspondence within a memory orbit. The enumeration in Q1 shows one distinction that a weight class alone cannot recover: the distribution of distances to the stored memories.

( We address this point in detail in our response to Q1 below )

W4. Minor

We agree that several modern Hopfield references in Section 2 have little direct bearing on this paper’s contribution. They give context for our choice to study the classical Hebbian model. The closer connections are to gauge symmetry in the weight transformation, Harrison’s work in the orbit count, and Hadamard equivalence in the recognition algorithm. We will shorten the broader survey.

AuthorsOpenReview ↗

Response to Reviewer Vy3t

Response to Questions

Q1. Don't the dynamics depend only on W, not on Ξ?

Yes. Under the update rule in the paper, WW determines every state transition. Two networks with the same WW therefore follow exactly the same trajectory from the same initial state. The weights alone, however, do not specify how close these states are to the memories used to construct the network. Consider these two memory matrices:

Ξ1=(−1−1−1−111−1−11−11−11−1−111111)\Xi_1= \begin{pmatrix} -1&-1&-1&-1\\\\ 1& 1&-1&-1\\\\ 1&-1& 1&-1\\\\ 1&-1&-1& 1\\\\ 1& 1& 1& 1 \end{pmatrix}

Ξ2=(−1−1−1111−111−1111−1−1−11111)\Xi_2= \begin{pmatrix} -1&-1&-1& 1\\\\ 1& 1&-1& 1\\\\ 1&-1& 1& 1\\\\ 1&-1&-1&-1\\\\ 1& 1& 1& 1 \end{pmatrix}

In each matrix the first four rows contribute 4I44I_4 to the Gram matrix, and the fifth row is the same. Setting the diagonal to zero gives

WΞ1=WΞ2=(0111101111011110)W_{\Xi_1}=W_{\Xi_2}= \begin{pmatrix} 0&1&1&1\\\\ 1&0&1&1\\\\ 1&1&0&1\\\\ 1&1&1&0 \end{pmatrix}

We enumerated all 16 initial states. For each state VV we measured the distance to the nearest stored row or its negative, using Hamming distance dHd_H:

\min_{1\le k\le M}\min\{d_H(V,\xi_k),\,d_H(V,-\xi_k)\}.$$ | Minimum distance | Initial states for $\Xi_1$ | Initial states for $\Xi_2$ | |---:|---:|---:| | 0 | 8 | 10 | | 1 | 8 | 6 | **The difference at distance zero has a simple explanation.** The first and fifth rows of $\Xi_1$ are negatives of each other, so its five rows represent four distinct pairs of the form $\{\xi,-\xi\}$. The five rows of $\Xi_2$ represent five such pairs. Signed row operations preserve these pairs up to ordering, and signed column operations preserve Hamming distance while relabeling the states. The histogram is therefore constant within a memory orbit. Its different values here distinguish two memory orbits within a single weight class. In future work we plan to determine which distinct memory orbits map to the same weight class under signed conjugation, and how much additional trajectory computation can be shared across those orbits.
Reviewer_Vy3tOpenReview ↗

Answer

Thank you for your clarification. While some concerns have been addressed, I'm still of the opinion that this work, as it is now, lacks the novelty needed for ICLR, and hence I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer Vy3t

Dear Reviewer Vy3t:

Thank you for the follow-up. We are glad our earlier reply clarified some of your concerns. We began by observing that different memory sets could give equivalent recall experiments after their targets and cues were put in correspondence. For the four transformations studied here, the familiar linear-algebraic relation describes how their weights transform. Our analysis goes beyond verifying that identity by proving that the correspondence holds through every nonlinear update, including at zero local field. This is what allows a recall experiment to be transferred between those memory sets.

Group theory gives this equivalence a quantitative meaning across the memory space. A memory set's internal symmetries determine how many different matrices belong to its orbit. Drawing matrices uniformly therefore samples each orbit in proportion to its size. A seemingly new candidate can repeat a recall problem already evaluated. The group description makes this repetition measurable by telling us how many matrices each representative stands for. The dynamical correspondence justifies one evaluation of the recall scores studied here per class. The orbit sizes supply the weights needed to recover the corresponding statistics over the full memory space.

The same connection between memory representation and retrieval suggests a direction for modern Hopfield networks. Here the group acts on whole memory sets. In a rotation-aware modern model, the different poses of one object can form an orbit representing a single memory. The corresponding retrieval should preserve object identity while the reconstructed state rotates with the cue. Establishing that the update and readout respect this action would extend the connection between memory equivalence and retrieval dynamics to that setting. We see this as a concrete direction for developing the group-based reasoning beyond the classical model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.