- Rating
- 3
- Confidence
- 2(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors present a robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates to achieve high-capacity storage of long sequences. Experiments demonstrate that synchronous asymmetric Hopfield networks possess substantial sequence-memory capacity, offering theoretical insights for both biological and artificial neural systems.
Critical strengths
- The paper addresses the joint challenge of obtaining many distinct, long-period limit-cycle attractors with noise robustness in a classical synchronous asymmetric Hopfield network.
- The block-cyclic architecture gives an intuitive explanation for the long periods: the labels on individual block cycles rotate, while the period of the full network is determined by the least common multiple of their lengths.
- The analysis extends from dense interblock connections to sparse connections and adversarial perturbations.
Critical weaknesses
- Most recovery experiments initialize all blocks in the (+1) state. These experiments assess denoising around a constant configuration but do not directly demonstrate retrieval of a distinct, nonconstant, long-period limit-cycle attractor.
- The construction relies on disjoint block cycles and synchronous updates, and the experiments use synthetic networks. The proposed relevance to biological circuits and sequence models would be more convincing with a clearer account of which conclusions might extend beyond these structural and update assumptions.
- The finite-size evaluation does not directly examine the claimed combination of attractor count, period, and retrieval robustness. Table 2 calculates capacity and period for sampled architectures, while the recovery simulations focus largely on constant block configurations.
- Given the paper’s claims about sequence memory and robust cyclic dynamics, a more focused discussion of recent closely related results would help clarify the contribution’s novelty and position.
Questions for authors
- Could you test noisy retrieval using several nonconstant, aperiodic block labelings?
- Which assumptions are essential for the capacity and robustness bounds?
3 comments
Thank you for your response, which has addressed some of my concerns. I will maintain my positive rating and increase my confidence from 1 to 2.
Thank you for the discussion so far. If you have any remaining concerns, please let us know; we are eager to address them.
We thank the reviewer for their careful review and constructive feedback.
Response to Weaknesses
This is a fair point. We have now conducted the retrieval experiments (Figure 2) with random aperiodic labelings and found the same outcome as with the constant start in every trial. With extra random connections (Figure 2(b)), we found that the network still recovers with aperiodic labelings, up to a few mismatched neurons per block, but the extra inputs no longer help. This is actually more consistent with our theory. We will add these runs and restrict the remark that extra inputs can help to the constant start in the revised version.
We agree that the scope should be stated more clearly. The dense, exactly disjoint structure is not essential: Theorems 2 and 3 keep similar guarantees when each neuron reads only three random neurons of the previous block, and when a small number of neurons per block receive arbitrary connections from anywhere in the network. Synchronous updates are essential to our mechanism, and whether the results extend to asynchronous dynamics is open, as noted in the conclusion. We will state this explicitly and make the remarks on biological circuits and sequence models more cautious.
For count and period, we enumerated all states of small networks, which confirms the count and period of the constructed attractors (see our answer to reviewer MY5e, Q2). For retrieval, the runs from aperiodic labelings described in our answer to W1 test the recovery of nonconstant, long-period attractors directly. All these results will be added to the revised version.
We will extend the related work (Section 2 and Table 4 in Appendix G.5) with the symmetric high-capacity constructions (Hillar & Tran, 2018; Fiete et al., 2014; Chaudhuri & Fiete, 2019), and sharpen the comparison with the statistical results on cycles in random asymmetric networks (Bastolla & Parisi, 1997; Hwang et al., 2019, 2020). We would welcome pointers to any other results you have in mind.
Response to Questions
Yes, we ran it from random aperiodic labelings and found the results to be consistent; see our answer to W1.
Essential are synchronous updates, the cyclic block structure in which each neuron reads the previous block, and blocks of at least order logn neurons for robustness. Not essential are dense connections (Theorem 2), exact disjointness of the cycles (Theorem 3) and the particular block size (Remark 2). The period bound needs many distinct cycle lengths, which random lengths provide; choosing them pairwise coprime does not improve it much (Remark 5).
References
Rishidev Chaudhuri and Ila Fiete. Bipartite expander Hopfield networks as self-decoding high-capacity error correcting codes, NeurIPS, 2019.
Ila Fiete, David Jason Schwab, Ngoc Mai Tran. A binary Hopfield network with 1/log(n) information rate and applications to grid cell decoding. arxiv, 2014
Christopher Hillar and Ngoc M. Tran. Robust exponential memory in Hopfield networks. Journal of Mathematical Neuroscience, 2018.
Sungmin Hwang, Viola Folli, Enrico Lanza, Giorgio Parisi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in asymmetric neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2019.
Sungmin Hwang, Enrico Lanza, Giorgio Parisi, Jacopo Rocchi, Giancarlo Ruocco, Francesco Zamponi. On the number of limit cycles in diluted neural networks. Journal of Statistical Physics, 2020.
U. Bastolla and G. Parisi. Attractors in fully asymmetric neural networks. Journal of Physics A: Mathematical and General, 1997.