- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper revisits spurious states in Hopfield networks, arguing that mixtures of stored patterns can be interpreted as multi-label signals rather than undesirable retrieval states. Using the Hopfield–RBM duality, the authors show that the hidden-layer activations encode the constituent patterns of a mixture. They derive an analytically specified label representation and prove that initializing RBM weights using class averages of noisy examples yields an asymptotic fixed point in expectation under supervised one-step contrastive divergence (CD-1), together with a finite-size bound on the expected update.
The authors extend the analysis to continuous signal mixtures using one-bit random projections and an arcsine overlap law. They evaluate the resulting method on four benchmarks spanning musical mixtures and hyperspectral data, showing that class-average initialization substantially outperforms random initialization and can reduce the number of required training steps. They also demonstrate that the resulting matched-filter classifier performs competitively against several sparse-recovery baselines at lower inference cost.
Critical strengths
S1. The paper provides a different perspective on the traditionally undesirable mixture states of Hopfield networks. The connection to the RBM hidden representation is intuitive, and Theorem 1 extends previous stationarity analyses to multi-label mixtures with a finite-size expected-drift bound. This is a meaningful theoretical contribution within the studied model.
S2. The authors evaluate their approach on musical mixtures and hyperspectral data, including settings that violate the assumptions of the theoretical model. Class-average initialization provides substantial improvements over random initialization across all four benchmarks, while the resulting matched-filter decoder achieves competitive performance against more computationally expensive reconstruction methods in several settings.
Critical weaknesses
W1. The empirical improvements do not convincingly establish the added value of the Hopfield–RBM theoretical framework.
-
W1.1. The primary training comparison in Figure 2 is between class-average initialization and random initialization. However, using class prototypes or supervised class averages is a standard way to initialize a classifier. It remains unclear whether the proposed construction offers an advantage over simpler prototype-based initialization methods.
-
W1.2. The experiments do not clearly separate the benefits of class-average weights, the theoretically derived hidden-label amplitudes, and the specific supervised CD-1 procedure. Consequently, it is difficult to determine which components of the framework are responsible for the reported improvements.
-
W1.3. Although the empirical results are promising, they do not establish that the Hopfield–RBM interpretation provides a meaningful practical improvement beyond established prototype-based classification approaches.
W2. The theoretical guarantees do not adequately explain the behavior observed on real-world signals.
-
W2.1. Strong modeling assumptions: Theorem 1 assumes independent Rademacher archetypes, independent corruption, and majority-vote mixtures. Real audio and hyperspectral signals violate these assumptions, as acknowledged in Section 3. The resulting finite-size drift bound is therefore not directly applicable to the main real-data experiments.
-
W2.2. The authors acknowledge that class-average weights are not stationary on real data and that training can substantially improve their performance. Furthermore, Appendix G.4 reports that the theoretical bound becomes vacuous under the measured real-data parameters.
-
W2.3. The theoretical analysis establishes an interesting property of an idealized model, but its ability to explain or predict the optimization dynamics and practical benefits of the proposed method remains insufficiently demonstrated.
Questions for authors
Q1: Can the authors compare the proposed method against simpler supervised prototype or nearest-centroid classifiers using the same representations and labeled examples? In particular, what benefit does the Hopfield–RBM formulation provide beyond using class-average templates for multi-label prediction?
Q2: Can the authors provide a more direct comparison between the predicted and observed CD-1 update behavior on real data? In particular, can they explain why the planted weights remain useful initializations even when the theoretical drift bound is vacuous and the weights are not stationary?
Q3: How does performance vary with unequal source amplitudes or strongly correlated archetypes? Figure 3 indicates that sparse-recovery methods substantially outperform the proposed decoder for some mixed hyperspectral pixels.
4 comments
W2.1, W2.2 and W2.3. We appreciate the reviewer catching this. The revised conclusion says that the guarantees are proved for Rademacher archetypes with site-independent corruption, that measured signals satisfy neither, that the bound is vacuous at the experimental parameters (Appendix G.4), and that the gradient moves the weights off the class averages, rebalancing the classes and decorrelating the archetypes. The abstract now states this last point too. Theorem 1 is a statement about the model. On real data it is a hypothesis that we test, not a guarantee.
Q2. Appendix G.3 measures the signed drift along the cue on real signals, for the three clamps. On chords at it is at , at and at the binary clamp. As a fraction of the positive phase that the negative phase cancels:
| positive phase cancelled | binary clamp | ||
|---|---|---|---|
| NSynth chords, | 56% | 44% | 22% |
| NSynth chords, | 51% | 40% | 15% |
| Urban spectra | 29% | 27% | 18% |
The theoretical amplitude cancels most in all three cases. The bound is vacuous because it is a worst case over all entries, and its dataset channel is of order one at the of measured data (Table 9). The measured drift is an average along the systematic direction and stays well below that. So the class averages are a good start without being a fixed point. Training moves them, mostly by rebalancing and decorrelating, as under W1.2, and a random start of the same scale needs hundreds to thousands of steps to reach the point where they begin (Figure 2).
Q3. Section 4.2 discusses the case you point to. The amplitude is derived for equal abundances. On Samson pixels shared by two materials the abundances are far from equal, and a faint material gives a smaller overlap than an absent one with a similar spectrum. Sparse recovery estimates the amplitudes, so it wins there, by in micro-F1, at the cost of about two orders of magnitude more arithmetic (Table 3). Correlated archetypes are dealt with in two places. Whitening brings the mean between the class codes on Urban from on the raw bands to (Appendix D.2). And Samson, with three strongly overlapping materials, is where training gains most (Section 4.1).
Thank you for the detailed rebuttal and the revisions. I appreciate the additional experiments and the authors' willingness to acknowledge the limitations of their framework.
However, my main concern (W1) remains unresolved. The new comparisons in Appendix E show that simpler prototype-based initializations achieve nearly identical performance, and Table 5 shows that the archetype filter consistently outperforms the proposed method in micro-F1. Please correct me if I misunderstood.
The new gradient analysis is useful, but the theoretically derived label amplitude does not lead to meaningful improvements in classification performance. Furthermore, the theoretical guarantees remain restricted to idealized assumptions, and the bound is vacuous on real data. The authors have clarified these limitations, but the gap between the theoretical contribution and the empirical improvements remains.
I therefore intend to maintain my rating of 2 (Weak Reject).
Thank you for your final assessment.
A brief clarification regarding your main point (W1): the "simpler prototype-based initializations" in Appendix E (Table 2) are not an independent alternative to our framework: they are precisely the Hebbian prescriptions induced by the Hopfield–RBM duality. The division by simply removes the multiplicative estimator bias required to state Theorem 1 cleanly, leaving the scale-invariant classifier unchanged.
What the duality contributes is not an arbitrary heuristic initialization, but the analytical joint specification of the weight scale () and the hidden clamping amplitudes ( / ) required for supervised CD-1 to balance its positive and negative phases.
As we show in Appendix G.3, these theoretical amplitudes successfully minimize the signed drift across physical signals.
We acknowledge that if the paper is judged strictly as an applied classification benchmark rather than a theoretical analysis of energy-based learning dynamics, these nuances of the training mechanism carry less weight.
Thank you again for your time and engagement with our work.
Thank you for a clear and fair assessment of the paper, and for helping us pinpoint where our contribution lies relative to standard prototype classifiers. We agree with your points on the gap between the model assumptions and real data (W2) and have revised the abstract and conclusion accordingly.
W1.1, W1.3 and Q1. At inference our read-out is a prototype classifier: each hidden unit correlates the query with the class average of its archetype. The revised Section 2 says so before the theory. The comparison with simpler prototype classifiers on the same representation is in Section 4.2 and Appendix F. All three filters there see the same sign code of the same query. The archetype filter correlates it with the projected class template, a nearest-centroid classifier built before the sign, and HD bundling with the majority vote of the class codes. Both tables evaluate the weights at step zero. With the order given the filters trade places from setting to setting (Table 4), and our class average is ahead on the deepest Urban mixtures. Under the parameter-free rule the archetype filter is ahead of ours in micro-F1 in every cell (Table 5). So the Hopfield-RBM view gives no advantage at inference, and we do not claim one.
The same holds for initialisation, which W1.1 asks about. Appendix E (Table 2) now runs supervised CD-1 from nine starting points under the protocol of Figure 2. Among them are three simpler templates: the plain class averages, the template of the archetype filter and the sign of the class averages. On Urban they start at 0.820, 0.843 and 0.832 and end at 0.851, 0.855 and 0.850, against 0.820 and 0.856 for ours; on the other settings they are as close. All of these are Hebbian prescriptions of the kind the duality with the Hopfield model suggests, one stored pattern per class, and they do equally well. A start that knows nothing about the classes needs time to catch up: from zero weights, 800 steps on Urban, 1200 on NSynth and 5600 on OpenMIC to reach the score the class averages have before training. On Samson 25 steps are enough, because there the class averages come from the spectral library and not from the pixels. Every start ends within 0.02 of the others.
What the duality adds, then, is on the training side. Using class averages to initialise a classifier is standard, as you say. What is not standard is knowing at which label amplitude supervised CD-1 leaves them nearly in place, and how much drift is left. The duality gives that amplitude, and on measured signals it is the clamp that leaves the smallest drift among those we tested, on chords of three and of five notes and on Urban spectra (Q2 below).
W1.2. The three components can now be read separately. For the weights, Table 2 shows that any class template gives the head start, and how long it lasts against starts of different scales. For CD-1, Appendix E takes apart what training changes on Urban: the class averages score 0.8227, training takes them to 0.8651, a single common gain leaves the score at 0.8227 exactly, and the six per-class gains alone give 0.8361. So about a third of the gain comes from rebalancing the classes and the rest from decorrelating the archetypes, while the spectra come back at a cosine of 0.99 or better. For the amplitude, Appendix G.3 measures the drift at fixed weights for the three clamps.