acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fixed Points: Superpolynomial Capacity of Asymmetric Hopfield Networks

Abstract

Classical Hopfield networks are limited to static patterns due to symmetric weights, whereas asymmetric networks can encode temporal sequences via limit-cycle attractors. Achieving high-capacity storage of long sequences in classical synchronous asymmetric networks, however, has remained a challenge. We present a simple and robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates, that allows neurons to support distinct limit-cycle attractors, each with period and robust to random noise with flip probability up to , yielding superpolynomial capacity in both the number and length of stored sequences. This is the first demonstration of such capacity for asymmetric Hopfield networks, which we obtain by combining results from combinatorics, number theory and the analysis of opinion dynamics. Our findings show that synchronous asymmetric Hopfield networks possess a sequence-memory capacity which is larger and more robust than previously recognized, demonstrating that, in both biological and artificial neural systems, robust sequence representation can be achieved through coarse architectural motifs rather than complex nonlinearities.

Primary area:
learning theory
Keywords:
Hopfield Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hym1OpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present a robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates to achieve high-capacity storage of long sequences. Experiments demonstrate that synchronous asymmetric Hopfield networks possess substantial sequence-memory capacity, offering theoretical insights for both biological and artificial neural systems.

Critical strengths
  1. The paper addresses the joint challenge of obtaining many distinct, long-period limit-cycle attractors with noise robustness in a classical synchronous asymmetric Hopfield network.
  2. The block-cyclic architecture gives an intuitive explanation for the long periods: the labels on individual block cycles rotate, while the period of the full network is determined by the least common multiple of their lengths.
  3. The analysis extends from dense interblock connections to sparse connections and adversarial perturbations.
Critical weaknesses
  1. Most recovery experiments initialize all blocks in the (+1) state. These experiments assess denoising around a constant configuration but do not directly demonstrate retrieval of a distinct, nonconstant, long-period limit-cycle attractor.
  2. The construction relies on disjoint block cycles and synchronous updates, and the experiments use synthetic networks. The proposed relevance to biological circuits and sequence models would be more convincing with a clearer account of which conclusions might extend beyond these structural and update assumptions.
  3. The finite-size evaluation does not directly examine the claimed combination of attractor count, period, and retrieval robustness. Table 2 calculates capacity and period for sampled architectures, while the recovery simulations focus largely on constant block configurations.
  4. Given the paper’s claims about sequence memory and robust cyclic dynamics, a more focused discussion of recent closely related results would help clarify the contribution’s novelty and position.
Questions for authors
  1. Could you test noisy retrieval using several nonconstant, aperiodic block labelings?
  2. Which assumptions are essential for the capacity and robustness bounds?
3 comments

We thank the reviewer for their careful review and constructive feedback.

Response to Weaknesses

  1. This is a fair point. We have now conducted the retrieval experiments (Figure 2) with random aperiodic labelings and found the same outcome as with the constant start in every trial. With extra random connections (Figure 2(b)), we found that the network still recovers with aperiodic labelings, up to a few mismatched neurons per block, but the extra inputs no longer help. This is actually more consistent with our theory. We will add these runs and restrict the remark that extra inputs can help to the constant start in the revised version.

  2. We agree that the scope should be stated more clearly. The dense, exactly disjoint structure is not essential: Theorems 2 and 3 keep similar guarantees when each neuron reads only three random neurons of the previous block, and when a small number of neurons per block receive arbitrary connections from anywhere in the network. Synchronous updates are essential to our mechanism, and whether the results extend to asynchronous dynamics is open, as noted in the conclusion. We will state this explicitly and make the remarks on biological circuits and sequence models more cautious.

  3. For count and period, we enumerated all states of small networks, which confirms the count and period of the constructed attractors (see our answer to reviewer MY5e, Q2). For retrieval, the runs from aperiodic labelings described in our answer to W1 test the recovery of nonconstant, long-period attractors directly. All these results will be added to the revised version.

  4. We will extend the related work (Section 2 and Table 4 in Appendix G.5) with the symmetric high-capacity constructions (Hillar & Tran, 2018; Fiete et al., 2014; Chaudhuri & Fiete, 2019), and sharpen the comparison with the statistical results on cycles in random asymmetric networks (Bastolla & Parisi, 1997; Hwang et al., 2019, 2020). We would welcome pointers to any other results you have in mind.

Response to Questions

  1. Yes, we ran it from random aperiodic labelings and found the results to be consistent; see our answer to W1.

  2. Essential are synchronous updates, the cyclic block structure in which each neuron reads the previous block, and blocks of at least order log⁡n\log n neurons for robustness. Not essential are dense connections (Theorem 2), exact disjointness of the cycles (Theorem 3) and the particular block size (Remark 2). The period bound needs many distinct cycle lengths, which random lengths provide; choosing them pairwise coprime does not improve it much (Remark 5).

References

  1. Rishidev Chaudhuri and Ila Fiete. Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes, NeurIPS, 2019.

  2. Ila Fiete, David Jason Schwab, Ngoc Mai Tran. A binary Hopfield network with 1/log⁡(n)1/ \log(n) information rate and applications to grid cell decoding. arxiv, 2014

  3. Christopher Hillar and Ngoc M. Tran. Robust exponential memory in Hopfield networks. Journal of Mathematical Neuroscience, 2018.

  4. Sungmin Hwang, Viola Folli, Enrico Lanza, Giorgio Parisi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in asymmetric neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2019.

  5. Sungmin Hwang, Enrico Lanza, Giorgio Parisi, Jacopo Rocchi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in diluted neural networks. Journal of Statistical Physics, 2020.

  6. U. Bastolla and G. Parisi. Attractors in fully asymmetric neural networks. Journal of Physics A: Mathematical and General, 1997.

Thank you for your response, which has addressed some of my concerns. I will maintain my positive rating and increase my confidence from 1 to 2.

Thank you for the discussion so far. If you have any remaining concerns, please let us know; we are eager to address them.

Reviewer_MY5eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper “Beyond fixed points: superpolynomial capacity of asymmetric Hopfield networks” investigate the retrieval capacity of asymmetric recurrent neural networks under synchronous update dynamics. In this regime attractors can be cycles of different lengths. Generally speaking both the number and the nature of attractors under synchronous dynamics depend on the realization of the weight matrix W. Previous works (e.g. Hwang et al. 2019, 2020) have characterized the case of asymmetric networks when W is randomly distributed. The current paper instead investigates the relation between the network topology (given all weights to be unitary) and the dynamics. The Authors find that only the topology can induce the emergence of an exponential number of cyclic attractors with connected finite entropy basins of attractions.

Critical strengths

The paper is mathematically solid and easy to read, even for researchers not from the theoretical computer science community. The tackled question is interesting for what concerns informatics, statistical physics and theoretical neuroscience. Specifically, the Authors investigate the relation between network topology and the complexity of cyclic attractors, while most of the literature focus on the implications of the nature of connections to dynamics. This work also analytically investigates the associativity of such limit cycles, i.e. the size of their basins of attraction. As far as I know, this point was only addressed numerically by Folli et al. (2018).

Critical weaknesses

The connection to Dale’s law of neuroscience is rather weak, since the topology of the network is anyways limited to unitary weights only, so the asymmetry is strongly related to a notion of dilution in the network. This aspect is certainly relevant in biology but the coexistence between excitatory and inhibitory interactions is also fundamental, as demonstrated by computational neuroscientists (i.e. Mongillo et al., 2018).
At first read the novelty with respect to previous works is a bit hidden. The authors should stress more, especially in the introduction, that they are answering a different question. For instance, what said at line 100, i.e. “We demonstrate that it is possible to achieve very large sequence memory capacity without altering the classical asymmetric Hopfield network model.”, is not accurate: previous works (e.g. Hwang et al., 2019) already show that classical asymmetric recurrent networks can reach an exponential number of limit cycles.

Questions for authors

I will now present my comments and questions with the goal of improving the quality of the presented work. I ask the Authors to address my points extensively.

Major issues:

Q1: The approach used in the analysis seems to first fix the topology of the network and then estimate the number and length of the limit cycles. Let us consider two extreme scenarios:

a) d=1d = 1, Nblocks=nN_{blocks} = n, then each neuron propagates the configuration of the only incoming neighbour. Since there are 2n2^n initializations and the max period is mm, i expect something like O(2n/m)O(2^n / m) cycles.

b) d=nd = n, Nblocks=1N_{blocks} = 1 i.e. the network is fully connected, but weights are all unitary, so i have only 2 point-wise attractors and 1 limit cycle of length 2.

It seems like the framework used by the paper can never tend to limit (b). It would be interesting to allow the model to reach the dense limit and define the scaling of the topological parameters such that the number of cycles ceases to be exponentially large but instead breaks and becomes polynomial. In this limit, the co-existence of excitatory and inhibitory interactions is indeed important to imply frustration and complexity of attractors. Also, Theorem 1 provides lower bounds for NcyclesN_{cycles} and the cycle period, but the scenario in (b) imposes an upper bound to exist as well, due to the density of connections increasing. I ask the Authors to expand around this point, whether the theory might be adapted.

Q2: May the authors at least try to simulate small size networks to validate Theorem 1, points 1 and 2? It should be hard, since one should enumerate the number of cycles brute-force. This limitation is never addressed along the paper.

Minor issues:

Q3: I would move Table 1 and Table 4 at the beginning of the Appendices.

5 comments

We thank the reviewer for their careful review and constructive feedback.

Response to Weaknesses

  1. You are right that all weights in our construction are positive and that its asymmetry comes from the direction of the connections, so Dale's principle only motivates the study of asymmetric networks and is not used by the construction. We agree that the coexistence of excitation and inhibition is fundamental (Mongillo et al., 2018). A construction with genuine excitatory and inhibitory populations is a natural extension, which we will mention in the revised Section 5.

  2. You are right that the sentence at line 100 is too broad: Hwang et al. (2019, 2020) already obtain exponentially many limit cycles in the classical model, but of fixed short length, in random networks with a prescribed symmetry, and without considering their basins of attraction. Our contribution is a superpolynomial number of cycles, each with a superpolynomial period, that are robust to noise. We will rephrase line 100 and make this distinction explicit in the introduction. We refer the reviewer to Section 2 and Table 4 in Appendix G.5 for further details.

Response to Questions

  1. Both extreme cases fall outside our regime: the construction always has many blocks, with dd at least Klog⁡2mK \log^2 m for sufficiently large K, and at most nεn^{\varepsilon} for some ε<1\varepsilon<1. In (a), with d=1d=1, your estimate is right in order, about 2n−O(n)2^{n-O(\sqrt n)} cycles (the period is the lcm of the cycle lengths rather than mm), but none of them is robust to noise. In (b), a single block is connected to itself, so for odd dd every state reaches one of the two constant states in one step. There is an upper bound: for odd dd, every attractor is constant on each block, so there are at most 2n/d2^{n/d} of them, which is polynomial once dd is of order n/log⁡nn/\log n. We thank the reviewer for pointing this out, we will add this as a remark.

  2. We ran this check: we enumerated all 2n2^n states of small dense networks, up to n=24n=24, and in every case the limit cycles from aperiodic labelings match the count in the proof of Theorem 1 and have period equal to the lcm of the cycle lengths. For odd dd, the enumeration also confirms that every attractor is constant on each block, which gives the upper bound in our answer to Q1. This will be added to the revised version.

  3. Thank you for the suggestion. We will move Tables 1 and 4 to the beginning of the appendix.

References

  1. Gianluigi Mongillo, Simon Rumpel, Yonatan Loewenstein. Inhibitory connectivity defines the realm of excitatory plasticity. Nature Neuroscience, 2018.

  2. Sungmin Hwang, Viola Folli, Enrico Lanza, Giorgio Parisi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in asymmetric neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2019.

  3. Sungmin Hwang, Enrico Lanza, Giorgio Parisi, Jacopo Rocchi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in diluted neural networks. Journal of Statistical Physics, 2020.

Reviewer_MY5eOpenReview ↗

Rebuttal

I thank the Authors for addressing all my questions clearly and extensively.

I am satisfied with the replies, and with the numerical effort to answer question Q2.

Regarding the discussion on the state of the art, I just do not agree, in the answer to W2, that Hwang et al. (2019) studies short cycles only. Section 3.5 of that paper, and the related appendix, shows that near full asymmetry there are limit cycles of arbitrary length in the thermodynamic limit, that implies the expected fully ergodic behaviour in that regime. I agree that basins of attraction, whether you can define them for a chaotic network, and being itself a hard question in the field, are not tackled.

I would ask the authors to upload, when possible, an updated version of the manuscript with the promised changes and discussions. This is allowed and suggested by the current conference.

Thank you for the follow-up and for the pointer to Section 3.5 of Hwang et al. (2019). You are right that our answer to W2 understated their results: near full asymmetry cycles of arbitrary length appear and the average cycle length diverges, while at full asymmetry the mean number of cycles grows only linearly in NN (their Sections 3.5 to 3.8). The revised introduction now says that random asymmetric networks have exponentially many short limit cycles and, at full asymmetry, about linearly many cycles that can be exponentially long, all counted without regard to their basins of attraction. What we add is a superpolynomial number of limit cycles that are at the same time superpolynomially long and robust to noise.

We have uploaded a revised version, with all changes in blue. The changes that answer your review are:

  • Introduction, first contribution bullet: the sentence at line 100 is replaced by the distinction from Hwang et al. described above (W2).
  • Section 5: Dale's law only motivates the study of asymmetric networks; the construction is purely excitatory, and networks with excitatory and inhibitory populations are mentioned as a natural extension (Mongillo et al., 2018) (W1).
  • New Remark 3: for odd block size dd every limit cycle is constant on each block, so there are at most 2n/d2^{n/d} of them, which is polynomial when dd is of order n/log⁡nn/\log n (Q1). The definition of block cycles now states that a block cycle of length 1 is a block connected to itself, as used in our answer to Q1(b).
  • New Appendix H.4: the exhaustive enumeration of small networks (Table 6) (Q2).
  • Tables 1 and 4 are now Tables 1 and 2, in a new Appendix A (Q3).

The changes made for the other reviewers are listed in a general comment.

Reviewer_7ScNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the capacity of classical asymmetric Hopfield networks to store temporal sequences as limit-cycle attractors. In particular, it shows the existence of asymmetric Hopfield networks with binary neurons and synchronous updates that support superpolynomial number of limit cycle attractors with superpolynomial period and robustness to random noise, in both dense and sparse connectivity cases. This suggests that rich and robust sequence memory can arise from coarse architectural structure rather than fine weight tuning or modified neuron models.

Critical strengths

The paper is well written and honest about the scope. It ties together combinatorics, number theory and distributed computing in a neat way. The insight that rich and robust sequence memory can arise from coarse architectural structure rather than fine weight tuning or modified neuron models is interesting.

Critical weaknesses
  • Symmetric binary Hopfield networks with exponentially many robust fixed points are known, e.g. [1], with counts comparable to or exceeding Claim (1) in Theorem 1. The novelty therefore lies in combining many attractors with superpolynomial periods, and only the period bound (2) requires asymmetry. I think the author should cite these previous works and discuss the novelty of the contribution in comparison to them.

  • Although Dale's law is used to motivate asymmetry, it seems to me that the construction requires only positive weights.

  • It would help to state what the constants in the results depend on, and in particular to give the explicit rate of o(1)o(1) terms, so that the readers know what network sizes are required.

[1] "Robust exponential memory in Hopfield networks", Hillar & Tran (2018), Journal of Mathematical Neuroscience

Questions for authors

How does capacity measured in bits compare with the symmetric disjoint-clique baseline? What (beyond the lcm trick) needs asymmetry?

2 comments

We thank the reviewer for their careful review and constructive feedback.

Response to Weaknesses

  1. We know this line of work well (Hillar & Tran, 2018; Fiete et al., 2014; Chaudhuri & Fiete, 2019), and we agree that the paper should discuss it. In fact our construction is the asymmetric counterpart of the disjoint-clique networks you mention in your question: each block stores one bit robustly, as a repetition code, and the asymmetric connections move these bits along the cycles. The number of limit-cycle attractors in Theorem 1 is exp⁡(Ω(n/log⁡2n))\exp(\Omega(n/\log^2 n)), while the robust construction of Hillar & Tran (2018, Theorem 2) stores about 22n2^{\sqrt{2n}} memories, so our count is the larger one. That said, your point is partly right. Many robust attractors alone do not require asymmetry: disjoint symmetric cliques of size dd already give 2n/d2^{n/d} robust fixed points, which is of the same order as our count (see our answer to your question below). What requires asymmetry is that the attractors are long sequences. Under synchronous updates a symmetric network only converges to fixed points or cycles of period two (Goles et al., 1985), so it cannot store sequences at all. Our paper provides a thorough comparison with existing results for asymmetric networks (Section 2 and Table 4 in Appendix G.5). We emphasize that our main contribution is demonstrating the ability to store a superpolynomial number of cycles, each with a superpolynomial period, while remaining robust to noise, without requiring modifications to the standard Hopfield dynamics or complicated topologies and weight-assignment rules. We will add a paragraph on this line of work to the related work, making this distinction explicit.

  2. You are right that the construction uses only positive weights: all neurons are excitatory, and the asymmetry comes from the direction of the connections. We fully acknowledge that Dale's principle was introduced solely to motivate the study of asymmetric connections and is not used by our construction. In the revised version we will state this explicitly in Section 5, and mention networks with both excitatory and inhibitory neurons as a natural extension.

  3. The failure probabilities for noise robustness are explicit in the proofs and written in asymptotic form only for simplicity: O(1/(nlog⁡2n))O(1/(n\log^2 n)) in Theorem 1 and O(n1−ε(1+ξ))O(n^{1-\varepsilon(1+\xi)}) in Theorem 2 (Appendix E). The constants c1,c2c_1, c_2, and the probability that the random construction attains the stated count and period, come from asymptotic number-theoretic results and are not explicit. For concrete sizes, Table 2 (Appendix G.4) evaluates the construction exactly at n=108n = 10^8: with block size d=800d = 800 it has about 103800010^{38000} limit cycles of period about 1010610^{106}, robust to flip probabilities up to 0.4. We will state the explicit failure terms in the theorems.

Response to Questions

  1. The disjoint-clique construction of Fiete et al. (2014) stores one bit per clique of order log⁡n\log n neurons, that is exp⁡(Ω(n/log⁡n))\exp(\Omega(n/\log n)) robust fixed points. Measured in bits, ours matches it to leading order: with blocks of size dd it carries n/d−O(n/d)n/d - O(\sqrt{n/d}) bits, and the argument of Remark 2 shows that blocks of the same size suffice for the same noise level; we will state this case explicitly in the remark. Beyond the lcm trick, what needs asymmetry is any period larger than two: under synchronous updates a symmetric network only reaches fixed points or 2-cycles, so even a single block cycle of length ℓ\ell, which stores a sequence of length ℓ\ell, cannot be realized symmetrically. The lcm trick then turns these periods into superpolynomial ones.

References

  1. Rishidev Chaudhuri and Ila Fiete. Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes, NeurIPS, 2019.

  2. Ila Fiete, David Jason Schwab, Ngoc Mai Tran. A binary Hopfield network with 1/log⁡(n)1/ \log(n) information rate and applications to grid cell decoding. arxiv, 2014

  3. Christopher Hillar and Ngoc M. Tran. Robust exponential memory in Hopfield networks. Journal of Mathematical Neuroscience, 2018.

  4. Eric Goles-Chacc, Françoise Fogelman-Soulié, Didier Pellegrin. Decreasing energy functions as a tool for studying threshold networks. Discrete Applied Mathematics, 1985.

I thank the authors for the clear response, which, together with the revised paper, addresses all my concerns. I therefore raised the score to 3. However my experience about the topic is quite limited, so I will keep the confidence score 2.

Comments on the paper

1 comment

Revised version uploaded

We have uploaded a revised version. All changes are in blue (citations are now green, so that blue marks only changed text).

  • Introduction (MY5e W2, hym1 W2). The first contribution bullet now states the difference from Hwang et al. (2019, 2020): their random networks have exponentially many short cycles and, at full asymmetry, about linearly many cycles that can be exponentially long, counted without regard to basins of attraction. The remarks on ML sequence models and on central pattern generators are more cautious.
  • Section 2 (7ScN W1, Q1; hym1 W4). New paragraph on robust high-capacity static memories in symmetric networks (Hillar & Tran, 2018; Fiete et al., 2014; Chaudhuri & Fiete, 2019), our dense construction as the asymmetric counterpart of the disjoint cliques, and why sequences need asymmetry (Goles et al., 1985). One sentence contrasts the statistical results on random networks with our construction.
  • Section 3. Theorem 1(3) states the failure probability, 1/(dn)=O(1/(nlog⁡2n))1/(dn)=O(1/(n\log^2 n)) (7ScN W3). Remark 2 adds the fixed-noise case: blocks of order log⁡n\log n, the clique size of Fiete et al., suffice, giving 2n/d−O(n/d)2^{n/d-O(\sqrt{n/d})} limit cycles (7ScN Q1). New Remark 3: for odd dd there are at most 2n/d2^{n/d} limit cycles (MY5e Q1). The definition of block cycles states that a cycle of length 1 is a block connected to itself.
  • Section 5. Dale's law only motivates the study of asymmetric networks; the construction is purely excitatory (7ScN W2, MY5e W1). Theorem 2(3) states the failure probability O(n1−ε(1+ξ))O(n^{1-\varepsilon(1+\xi)}) (7ScN W3).
  • Section 6 and new Appendix H.4 (hym1 W1, W3, Q1; MY5e Q2). Retrieval from random aperiodic labelings on the same graphs and flips as Figure 2 (Table 5), and exhaustive enumeration of small networks (Table 6). The remark that extra inputs can help is restricted to the constant start.
  • Conclusion (hym1 W2, Q2). Which assumptions the mechanism relies on, and which it does not need.
  • Appendix A (MY5e Q3; hym1 W4). The notation table and the related-work table are now Tables 1 and 2 at the start of the appendix; Table 2 includes the symmetric constructions.

Numbering: the remarks after Remark 2 and the appendices shift by one (the former Appendices A to G are now B to H). In the numbers cited in our replies, Table 4 (Appendix G.5) is now Table 2 (Appendix A.2), Table 2 (Appendix G.4) is now Table 3 (Appendix H.3), Remark 5 is now Remark 6, and the proofs of Appendix E are now in Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.