acceptodds
Under review as a conference paper at ICLR 2027

Spurious states are labels: A multi-label code in the Hopfield-RBM duality

Abstract

For forty years, the superpositions of stored patterns known as spurious states have been treated as the defining defect of the Hopfield model. We read them as an answer: a mixture of patterns is a composite signal, and identifying its constituents is multi-label classification. A Hopfield configuration cannot represent a set of labels; its dual restricted Boltzmann machine (RBM) can. We show that the Hopfield-RBM duality analytically fixes this representation: across visible neurons, active labels sit on the hidden layer at scale with an amplitude matching the archetype-mixture overlap. Under this label assignment, the class averages of noisy training examples form an exact fixed point in mean of one-step supervised contrastive divergence. We prove a finite-size drift bound that decomposes into four vanishing error channels: thermal sampling, dataset noise, mini-batch consensus, and finite-dimensional fluctuations. For continuous mixtures combined prior to quantization (such as musical chords and hyperspectral pixels), this overlap amplitude transitions to an arcsine law via one-bit random projections. Across four benchmarks on three public datasets, the resulting configuration acts as an optimal analytical warm start: computed in a single pass without gradient updates, it surpasses random initializations by to in score, saving thousands of training steps. Finally, the induced parameter-free read-out rivals computationally intensive one-bit compressed sensing baselines at a fraction of their inference cost.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
statistical physics, associative memories, Hopfield networks, restricted Boltzmann machines, multi-label classification, contrastive divergence, energy-based models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BkxqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits spurious states in Hopfield networks, arguing that mixtures of stored patterns can be interpreted as multi-label signals rather than undesirable retrieval states. Using the Hopfield–RBM duality, the authors show that the hidden-layer activations encode the constituent patterns of a mixture. They derive an analytically specified label representation and prove that initializing RBM weights using class averages of noisy examples yields an asymptotic fixed point in expectation under supervised one-step contrastive divergence (CD-1), together with a finite-size bound on the expected update.

The authors extend the analysis to continuous signal mixtures using one-bit random projections and an arcsine overlap law. They evaluate the resulting method on four benchmarks spanning musical mixtures and hyperspectral data, showing that class-average initialization substantially outperforms random initialization and can reduce the number of required training steps. They also demonstrate that the resulting matched-filter classifier performs competitively against several sparse-recovery baselines at lower inference cost.

Critical strengths

S1. The paper provides a different perspective on the traditionally undesirable mixture states of Hopfield networks. The connection to the RBM hidden representation is intuitive, and Theorem 1 extends previous stationarity analyses to multi-label mixtures with a finite-size expected-drift bound. This is a meaningful theoretical contribution within the studied model.

S2. The authors evaluate their approach on musical mixtures and hyperspectral data, including settings that violate the assumptions of the theoretical model. Class-average initialization provides substantial improvements over random initialization across all four benchmarks, while the resulting matched-filter decoder achieves competitive performance against more computationally expensive reconstruction methods in several settings.

Critical weaknesses

W1. The empirical improvements do not convincingly establish the added value of the Hopfield–RBM theoretical framework.

  • W1.1. The primary training comparison in Figure 2 is between class-average initialization and random initialization. However, using class prototypes or supervised class averages is a standard way to initialize a classifier. It remains unclear whether the proposed construction offers an advantage over simpler prototype-based initialization methods.

  • W1.2. The experiments do not clearly separate the benefits of class-average weights, the theoretically derived hidden-label amplitudes, and the specific supervised CD-1 procedure. Consequently, it is difficult to determine which components of the framework are responsible for the reported improvements.

  • W1.3. Although the empirical results are promising, they do not establish that the Hopfield–RBM interpretation provides a meaningful practical improvement beyond established prototype-based classification approaches.

W2. The theoretical guarantees do not adequately explain the behavior observed on real-world signals.

  • W2.1. Strong modeling assumptions: Theorem 1 assumes independent Rademacher archetypes, independent corruption, and majority-vote mixtures. Real audio and hyperspectral signals violate these assumptions, as acknowledged in Section 3. The resulting finite-size drift bound is therefore not directly applicable to the main real-data experiments.

  • W2.2. The authors acknowledge that class-average weights are not stationary on real data and that training can substantially improve their performance. Furthermore, Appendix G.4 reports that the theoretical bound becomes vacuous under the measured real-data parameters.

  • W2.3. The theoretical analysis establishes an interesting property of an idealized model, but its ability to explain or predict the optimization dynamics and practical benefits of the proposed method remains insufficiently demonstrated.

Questions for authors

Q1: Can the authors compare the proposed method against simpler supervised prototype or nearest-centroid classifiers using the same representations and labeled examples? In particular, what benefit does the Hopfield–RBM formulation provide beyond using class-average templates for multi-label prediction?

Q2: Can the authors provide a more direct comparison between the predicted and observed CD-1 update behavior on real data? In particular, can they explain why the planted weights remain useful initializations even when the theoretical drift bound is vacuous and the weights are not stationary?

Q3: How does performance vary with unequal source amplitudes or strongly correlated archetypes? Figure 3 indicates that sparse-recovery methods substantially outperform the proposed decoder for some mixed hyperspectral pixels.

4 comments

Thank you for a clear and fair assessment of the paper, and for helping us pinpoint where our contribution lies relative to standard prototype classifiers. We agree with your points on the gap between the model assumptions and real data (W2) and have revised the abstract and conclusion accordingly.

W1.1, W1.3 and Q1. At inference our read-out is a prototype classifier: each hidden unit correlates the query with the class average of its archetype. The revised Section 2 says so before the theory. The comparison with simpler prototype classifiers on the same representation is in Section 4.2 and Appendix F. All three filters there see the same sign code of the same query. The archetype filter correlates it with the projected class template, a nearest-centroid classifier built before the sign, and HD bundling with the majority vote of the class codes. Both tables evaluate the weights at step zero. With the order given the filters trade places from setting to setting (Table 4), and our class average is ahead on the deepest Urban mixtures. Under the parameter-free rule the archetype filter is ahead of ours in micro-F1 in every cell (Table 5). So the Hopfield-RBM view gives no advantage at inference, and we do not claim one.

The same holds for initialisation, which W1.1 asks about. Appendix E (Table 2) now runs supervised CD-1 from nine starting points under the protocol of Figure 2. Among them are three simpler templates: the plain class averages, the template of the archetype filter and the sign of the class averages. On Urban they start at 0.8200.820, 0.8430.843 and 0.8320.832 and end at 0.8510.851, 0.8550.855 and 0.8500.850, against 0.8200.820 and 0.8560.856 for ours; on the other settings they are as close. All of these are Hebbian prescriptions of the kind the duality with the Hopfield model suggests, one stored pattern per class, and they do equally well. A start that knows nothing about the classes needs time to catch up: from zero weights, 800800 steps on Urban, 12001200 on NSynth and 56005600 on OpenMIC to reach the score the class averages have before training. On Samson 2525 steps are enough, because there the class averages come from the spectral library and not from the pixels. Every start ends within 0.020.02 of the others.

What the duality adds, then, is on the training side. Using class averages to initialise a classifier is standard, as you say. What is not standard is knowing at which label amplitude supervised CD-1 leaves them nearly in place, and how much drift is left. The duality gives that amplitude, and on measured signals it is the clamp that leaves the smallest drift among those we tested, on chords of three and of five notes and on Urban spectra (Q2 below).

W1.2. The three components can now be read separately. For the weights, Table 2 shows that any class template gives the head start, and how long it lasts against starts of different scales. For CD-1, Appendix E takes apart what training changes on Urban: the class averages score 0.82270.8227, training takes them to 0.86510.8651, a single common gain leaves the score at 0.82270.8227 exactly, and the six per-class gains alone give 0.83610.8361. So about a third of the gain comes from rebalancing the classes and the rest from decorrelating the archetypes, while the spectra come back at a cosine of 0.990.99 or better. For the amplitude, Appendix G.3 measures the drift at fixed weights for the three clamps.

W2.1, W2.2 and W2.3. We appreciate the reviewer catching this. The revised conclusion says that the guarantees are proved for Rademacher archetypes with site-independent corruption, that measured signals satisfy neither, that the bound is vacuous at the experimental parameters (Appendix G.4), and that the gradient moves the weights off the class averages, rebalancing the classes and decorrelating the archetypes. The abstract now states this last point too. Theorem 1 is a statement about the model. On real data it is a hypothesis that we test, not a guarantee.

Q2. Appendix G.3 measures the signed drift along the cue on real signals, for the three clamps. On chords at L=3L=3 it is 0.1710.171 at gLg_L, 0.2790.279 at cLc_L and 0.7790.779 at the binary clamp. As a fraction of the positive phase that the negative phase cancels:

positive phase cancelledgLg_LcLc_Lbinary clamp
NSynth chords, L=3L=356%44%22%
NSynth chords, L=5L=551%40%15%
Urban spectra29%27%18%

The theoretical amplitude cancels most in all three cases. The bound is vacuous because it is a worst case over all NKNK entries, and its dataset channel is of order one at the ρ\rho of measured data (Table 9). The measured drift is an average along the systematic direction and stays well below that. So the class averages are a good start without being a fixed point. Training moves them, mostly by rebalancing and decorrelating, as under W1.2, and a random start of the same scale needs hundreds to thousands of steps to reach the point where they begin (Figure 2).

Q3. Section 4.2 discusses the case you point to. The amplitude gLg_L is derived for equal abundances. On Samson pixels shared by two materials the abundances are far from equal, and a faint material gives a smaller overlap than an absent one with a similar spectrum. Sparse recovery estimates the amplitudes, so it wins there, by 0.2120.212 in micro-F1, at the cost of about two orders of magnitude more arithmetic (Table 3). Correlated archetypes are dealt with in two places. Whitening brings the mean ∣cos⁡∣|\cos| between the class codes on Urban from 0.6390.639 on the raw bands to 0.0240.024 (Appendix D.2). And Samson, with three strongly overlapping materials, is where training gains most (Section 4.1).

Thank you for the detailed rebuttal and the revisions. I appreciate the additional experiments and the authors' willingness to acknowledge the limitations of their framework.

However, my main concern (W1) remains unresolved. The new comparisons in Appendix E show that simpler prototype-based initializations achieve nearly identical performance, and Table 5 shows that the archetype filter consistently outperforms the proposed method in micro-F1. Please correct me if I misunderstood.

The new gradient analysis is useful, but the theoretically derived label amplitude does not lead to meaningful improvements in classification performance. Furthermore, the theoretical guarantees remain restricted to idealized assumptions, and the bound is vacuous on real data. The authors have clarified these limitations, but the gap between the theoretical contribution and the empirical improvements remains.

I therefore intend to maintain my rating of 2 (Weak Reject).

Thank you for your final assessment. A brief clarification regarding your main point (W1): the "simpler prototype-based initializations" in Appendix E (Table 2) are not an independent alternative to our framework: they are precisely the Hebbian prescriptions induced by the Hopfield–RBM duality. The division by r^\hat{r} simply removes the multiplicative estimator bias required to state Theorem 1 cleanly, leaving the scale-invariant classifier unchanged.
What the duality contributes is not an arbitrary heuristic initialization, but the analytical joint specification of the weight scale (N\sqrt{N}) and the hidden clamping amplitudes (cLc_L / gLg_L) required for supervised CD-1 to balance its positive and negative phases. As we show in Appendix G.3, these theoretical amplitudes successfully minimize the signed drift across physical signals.
We acknowledge that if the paper is judged strictly as an applied classification benchmark rather than a theoretical analysis of energy-based learning dynamics, these nuances of the training mechanism carry less weight. Thank you again for your time and engagement with our work.

Reviewer_Lxj8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Classical Hopfield networks retrieve a state that may correspond to an individual stored pattern or a mixture of patterns, but they do not directly identify the constituent patterns. The authors exploit Hopfield–RBM duality to represent each stored pattern through a separate hidden unit, whose activation measures its correlation with the input mixture. These activations can then be thresholded to produce multiple labels. Since the RBM hidden units are continuous, the authors derive a mixture-dependent target activation scale and show that this choice makes class-average weights approximately stationary under supervised contrastive-divergence learning.

Critical strengths
  1. The paper offers an important perspective on spurious states where the overlap between states with several stored patterns can help identify the components of a mixture.

  2. The theoretical analysis in the paper separates four sources of learning drift. Controlled experiments are performed to examine each source individually, and the result connect the bound to observable behavior.

  3. The experiments include both constructed mixtures and measured signals. The authors also report important limitations, including poor decoding results and cases where the theoretical predictions do not hold.

Critical weaknesses

Major Weaknesses

  1. The paper is difficult to follow on how it connects the theoretical analysis to the proposed multi-label classification method. The underlying mechanism is relatively straightforward: class-average prototypes are used as RBM weights, the hidden activations measure correlations between the input mixture and the stored patterns, and a threshold is performed on these correlations to identify the constituent labels. The authors should explain the central idea and the role of the RBM more clearly before introducing the theoretical analysis.

  2. The theoretical advance over prior work needs a clearer explanation. Leonelli et al. (2021) already study class-average initialization and its fixed-point properties. Remark 2 identifies the multi-label setting and finite-parameter bounds as extensions, but the paper should explain what additional insight these provide about classification or learning. The connection to earlier work on compositional RBM representations, such as Tubiana and Monasson (2017), also deserves a clearer discussion.

  3. The evaluation lacks comparisons with Modern Hopfield Networks and their variants. MHNs can retain states that average subsets of patterns, and sparse and structured variants provide relevant alternatives (Ramsauer et al., 2021; Hu et al., 2023; Santos et al., 2024). With the same prototypes and input, a single softmax attention step preserves the correlation ranking and therefore gives the same top-L labels. A controlled comparison would help establish what the proposed training and decoding methods add, especially when the number of labels is unknown.

  4. The stationarity claim needs correction. Theorem 1 bounds the expected update but does not establish that it is exactly zero at finite parameters. Even one noiseless pattern has a nonzero expected update at finite temperature. The limiting argument also needs to specify how the learning rate scales with inverse temperature.

Minor Weaknesses

  1. The paper compares class-average initialization with random weights but does not evaluate other initialization methods. The abstract’s claim of an "optimal analytical warm start" therefore needs stronger support.

  2. The strongest decoding results assume that the number of labels is known. Figure 3 supplies this information to every method. Without it, NSynth micro-F1 for five-source mixtures falls from 0.627 to 0.397. The archetype filter also achieves higher micro-F1 in every setting in Table 4, although it uses prototype information from before binarization.

  3. The experiments do not show a clear classification benefit from the theoretical label amplitude. For three-source NSynth mixtures, the update measured along the input direction reaches zero near 0.219, compared with the theoretical amplitude of 0.392. The theoretical and simpler amplitudes also produce micro-F1 scores within 0.005 of each other during training. The authors should clarify what practical improvement depends on the proposed amplitude.

Questions for authors
  1. How should the fixed-point claim be revised given the nonzero expected update at finite temperature? How must the learning rate scale for the limiting result to hold?

  2. What do the theoretical results add beyond Leonelli et al. (2021), and what new insight do they provide about multi-label classification?

  3. How does the method compare with Modern Hopfield Networks and their sparse or structured variants when the same prototypes, inputs, and label-selection rules are used?

  4. How does class-average initialization compare with other initialization methods using the same training data?

  5. Can the proposed decoder match or outperform the archetype filter when the number of active labels is unknown?

  6. What classification benefit does the theoretical label amplitude provide, given the nearly identical results with simpler choices?

  7. How do accuracy and total inference time compare with a direct filter on the real-valued features, including the cost of encoding and decoding?

5 comments

Thank you for a remarkably thorough and constructive review. Your questions helped us disentangle the roles of inference and training and motivated three new sets of experiments (W3, W5, and W6).

W1. We agree, and we used your own summary. A new paragraph in Section 2, before Definition 1 and Theorem 1, separates three roles. The weights are set to the class averages, one column per archetype. At fixed weights, hidden unit μ\mu returns the overlap between the mixture and its class average. This is a matched filter, and the label amplitude plays no role in it. The rule that turns the overlaps into a set of labels is the parameter-free threshold of Section 3. The RBM matters for training: its hidden layer is where the label is clamped, and the theory fixes the scale and the amplitude of that clamp.

W4 and Q1. You are right. At finite β\beta the expected update is not zero, already for one noiseless pattern; this is the thermal channel of the bound. We have withdrawn the exact claim. Definition 1 now defines δ\delta-stationarity in mean, ∣E[Δw]∣/(ϵβ)≤δ|\mathbb E[\Delta w]|/(\epsilon\beta)\le\delta, and asymptotic stationarity as δ→0\delta\to0. Theorem 1 bounds δ\delta at finite NN, MM, BB and β\beta, and the normalised drift vanishes as these grow, with ln⁡(KT)=o(N)\ln(KT)=o(N), in any order. On the learning rate: for the update itself to vanish, ϵβ\epsilon\beta must stay bounded, for instance ϵ\epsilon of order 1/β1/\beta. Algorithm 1 is stated with this assumption.

W2 and Q2. Remark 2 now answers this. Leonelli et al. plant the plain sample mean and show, for one archetype at a time, that the mean update vanishes in a joint limit. Theorem 1 differs in three ways. The target is a set of LL labels on a single hidden layer, which forces the clamp NcL\sqrt N c_L and gives back their unit clamp only at L=1L=1. The weights are debiased by rr, so that one number ρ\rho carries the imperfection of the data. And the bound holds at finite parameters, which separates four channels. What this says about multi-label learning is that a multi-hot label at unit amplitude is the wrong target: the positive phase then differs from what the negative phase returns at the class averages by (1−cL)N(1-c_L)\sqrt N, and the difference survives every limit (Remark 1). The ordering holds on measured chords too, where no hypothesis of the theorem does: at L=3L=3 the drift is 0.1710.171 at gLg_L, 0.2790.279 at cLc_L and 0.7790.779 at the binary clamp (Appendix G.3). On Tubiana & Monasson (2017), the introduction now presents our setting as the supervised counterpart of their compositional regime. There composition is learned; here it is imposed by the task and read off the hidden layer.

W3 and Q3. We would like to thank the referee for raising this point. We ran the comparison with a Modern Hopfield Network, and it is now Appendix F.1. We gave a modern Hopfield network exactly what the machine has: the class averages as stored patterns and the same sign code as input. One attention step then gives p=f(βZ)\boldsymbol p=f(\beta\boldsymbol Z), where Z\boldsymbol Z are the overlaps of the machine and ff is the softmax, the sparsemax of Hu et al. (2023) or the 1.5-entmax of Santos et al. (2024). As you say, with LL given all of these return the same top-LL labels as the machine. So the question is what happens when LL is unknown, where each needs a rule that turns p\boldsymbol p into a set. We tried our largest-gap rule on the softmax, a threshold on the softmax, and the support of sparsemax and entmax. β\beta and the threshold are chosen on validation draws with seeds not used for testing, one value for all orders. Tuning is a help our parameter-free rule does not get, so we also gave the machine a threshold tuned in the same way, directly on Z\boldsymbol Z and with no attention. Micro-F1 pooled over the orders, mean over the test seeds:

read-outUrbanSamsonOpenMICNSynth
machine, largest gap (no parameters)0.8000.6960.5670.495
softmax, largest gap0.7920.7250.5690.491
softmax, tuned threshold0.8360.7650.5730.595
sparsemax support, tuned β\beta0.8220.7480.5680.558
1.5-entmax support, tuned β\beta0.8100.7590.5680.509
machine, tuned threshold0.8570.7730.5670.611

With the same cut rule, attention changes little: the largest gap gives within 0.030.03 of the machine whether it is applied to Z\boldsymbol Z or to its softmax. With tuning, the modern Hopfield read-outs pull ahead of our parameter-free rule, by up to 0.100.10 on NSynth. But the machine with a threshold tuned in the same way does better than all of them, in every test seed on Urban, Samson and NSynth, and is within 0.0060.006 on OpenMIC. So the attention step adds nothing to the overlaps. What helps when LL is unknown is a cut learned from labelled data, and the machine gains from it as much as any attention variant.

The experiment also shows what our parameter-free rule costs. In deep mixtures the constituents do not sit at one common level, and the largest gap often falls between two of them. At L=5L=5 on NSynth it returns two labels on average where the tuned threshold returns four. Appendix F.1 reports this next to the table.

W5 and Q4. Other initialisations. We removed "optimal" from the abstract, and we ran the comparison you ask for (Appendix E, Table 2). The protocol is that of Figure 2, with the same seeds, training data and learning rates, and nine starting points. Four of them are class templates: our class averages, the plain averages not divided by r^\hat r (Leonelli et al.), the template of the archetype filter, built before the sign, and the sign of the class average, which is what HD bundling stores. The others know nothing about the classes: zero weights, and Gaussian weights of several scales, one of which is the random start of Figure 2. The table gives the score before training and after 8⋅1038\cdot10^3 steps at the best learning rate and, in brackets, the steps needed to reach the score our class averages have before training.

starting pointNSynthUrbanSamsonOpenMIC
class averages over r^\hat r (ours)0.484 →\to 0.5030.820 →\to 0.8560.698 →\to 0.8540.534 →\to 0.529
plain class averages0.484 →\to 0.4990.820 →\to 0.8510.698 →\to 0.8520.534 →\to 0.533
archetype template0.508 →\to 0.5120.843 →\to 0.8550.709 →\to 0.8520.542 →\to 0.535
sign of the class averages0.480 →\to 0.5010.832 →\to 0.8500.701 →\to 0.8390.532 →\to 0.524
zero weights0 →\to 0.497 (1200)0 →\to 0.848 (800)0 →\to 0.839 (25)0.447 →\to 0.536 (5600)
Gaussian, scale of the weights0.108 →\to 0.499 (1600)0.389 →\to 0.845 (1200)0.496 →\to 0.838 (100)0.454 →\to 0.525 (never)
Gaussian, three times that0.113 →\to 0.492 (3200)0.395 →\to 0.843 (1600)0.483 →\to 0.836 (400)0.447 →\to 0.516 (never)

The first and the sixth row are a rerun of Figure 2, and agree with it to within 0.0030.003. On Samson the sixth row reads 100100 steps where Figure 2 gives 4⋅1024\cdot10^2: the random start is the same, but here the score is also read after 1010, 2525, 5050, 100100 and 200200 steps, while Figure 2 reads it every 400400.

The four templates behave alike. They start within 0.030.03 of each other and end within 0.020.02. All of them are Hebbian prescriptions of the kind the duality with the Hopfield model suggests, one stored pattern per class built from the examples of that class, so we read this as support for the duality as a source of good starting points rather than for one particular variant. Dividing by r^\hat r only rescales the weights, which the decoder does not see; it is needed for the statement of Theorem 1, not for the score.

Every start ends within 0.020.02 of every other. The class averages are a head start, not a better end point, as Section 4.1 already says. How long the head start is depends on what it is measured against. Starting from zero weights takes 800800 steps on Urban, 12001200 on NSynth and 56005600 on OpenMIC to reach the score the class averages have before any training. Samson is the exception: 2525 steps are enough, because there the class averages come from the spectral library and not from the pixels, and a handful of labelled pixels already does as well.

Reviewer_QozuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper reads the spurious (mixture) states of the Hopfield model as multi-label queries: given a superposition of stored patterns, name the constituents. In the dual RBM, each hidden unit corresponds to one pattern, and labels are written at the classical mixture-overlap amplitude. Theorem 1 shows that debiased class averages of noisy examples are a fixed point in mean of supervised one-step contrastive divergence, up to four vanishing error terms. For real signals, a sign-of-random-projection encoder turns the amplitude into an arcsine law. Experiments on NSynth, Urban/Samson and OpenMIC show a better starting point than random initialization and a cheap read-out competitive with one-bit compressed sensing.

Critical strengths

The theory is internally consistent and checked carefully on synthetic data where its assumptions hold.

Critical weaknesses
  1. The motivation is weak, and the most relevant recent work is missing.
  • The paper starts from the Hopfield spurious state and looks for a task it can be mapped to, rather than starting from a problem that needs this tool. Chord recognition and hyperspectral unmixing have established methods, and the paper never compares against them.
  • The three literatures in the introduction are linked only by analogy. There is no SAE experiment, and HD bundling performs the same as the proposed read-out.
  • The opening claim that spurious states have been treated purely as defects overlooks recent work that already recasts them as useful.
  • Most notably, Pham et al. (2025) identify spurious states as the first signs of generative capability in diffusion models, at the transition from memorization to generalization. This is a far more consequential setting, and the paper doesn’t cite it.
  1. Stripped of the physics, the method is template matching, and the theory doesn’t describe the data.
  • The read-out is the inner product between a query’s random sign code and each class’s average code. HD bundling stays within about 0.02 of it, and the plain archetype filter beats it in every cell under the parameter-free rule (Table 4).
  • Lemmas 1–2 are classical. Theorem 1’s fixed point holds largely by construction, since the label amplitude is set to the response the planted weights produce.
  • On real data: 1) only about 6 sites out of 16,384 are effectively independent; 2) the bound is vacuous (17–166× the trivial scale); 3) CD-1 moves the weights away from the planted point.
  • Absolute performance is weak. OpenMIC macro-AP is 0.534 against a chance level of 0.447, and on Samson at L=2 the method falls below “predict every class” (0.558 vs 0.800).
Questions for authors
  1. Why use an RBM for chord recognition or hyperspectral unmixing at all? How does the method compare to standard approaches on the real-valued signals, such as non-negative least-squares unmixing?
  2. Beyond HD bundling and the archetype filter, what does the Hopfield–RBM view add in practice?
  3. How does this work relate to Pham et al. (2025), which already treats spurious states as useful rather than as defects?
5 comments

Thank you for a candid and challenging review. We agree with several of your points and say so below and led us to add the least-squares baseline and expand our discussion of the literature; on one point, whether Theorem 1 holds by construction, we respectfully disagree.

W1 and Q3. Thank you for the reference of Pham et al. (2025). The paper is now cited in the introduction, in its published version (Nature Communications, 2026), and the opening no longer says that spurious states have only been treated as defects. The two results complement each other. Pham et al. find spurious states at the transition from memorization to generalization, that is, once the memory is overloaded. We work below capacity, at K=O(1)K=O(1), where the mixtures are the classical ones of Amit, Gutfreund and Sompolinsky, and ask what the dual RBM can read from them. The paragraph on dense associative memories now notes that these suppress mixtures below capacity and that mixtures reappear once the memory is overloaded, as Pham et al. show.

W1. The question is what a spurious state is in the dual RBM, and the answer is a multi-label query. The datasets are test beds where the superposition is physical, a chord or a pixel, so that the hypotheses of the theory can be checked one at a time. They are not offered as applications in which the RBM competes with the established methods.

W1 and Q1. Established methods, such as non-negative least squares.** We ran it (Appendix F.2, Table 7). On Urban and Samson it is fully constrained least squares on the real-valued spectra, with the class means of the prototype spectra as endmembers. On NSynth and OpenMIC it is plain non-negative least squares on the features, with the class means as dictionary. The queries and seeds are those of Tables 4 and 5. Micro-F1 with the order given, one value per order:

settingthe machine, one-bit codesleast squares, real-valued signal
Urban, L=1,…,5L=1,\dots,50.951 / 0.908 / 0.886 / 0.891 / 0.9150.888 / 0.849 / 0.848 / 0.868 / 0.912
Samson, L=1,2L=1,20.981 / 0.5670.979 / 0.715
OpenMIC, L=1,2L=1,20.560 / 0.6530.439 / 0.547
NSynth, L=1,2,3,5L=1,2,3,50.755 / 0.625 / 0.602 / 0.6270.755 / 0.683 / 0.672 / 0.694

Neither wins everywhere. The machine is ahead on Urban up to L=4L=4 and level at L=5L=5, and ahead on OpenMIC. Least squares is ahead on the Samson pixels that two materials share and on chords of two notes or more, and the two are level on single sources there. With the order unknown and the largest-gap rule on both, the picture is similar (Table 7). Least squares wins where the abundances are unequal, because it estimates the amplitudes that our read-out takes to be equal; this is the same reason the sparse reconstructions of Section 4.2 win on those pixels. It also reads the whole real-valued signal and solves a constrained problem for each query, where the machine reads NN bits and takes one inner product per class. We agree that an RBM is not the tool one would pick for hyperspectral unmixing, and the paper does not claim it is.

W1. The three literatures, SAEs and HD bundling. The revised introduction now discusses the compositional regime of RBMs (Tubiana & Monasson, 2017), of which our setting is the supervised counterpart, and mentions sparse autoencoders only as the unsupervised version of the same decoding problem. We agree that the link to SAEs is an analogy and nothing more. There is no SAE experiment, and we do not claim a result about SAEs. That HD bundling stays close to our read-out is expected: both are matched filters on the same codes, and they differ only in whether the class code is averaged or majority-voted (Section 4.2).

W2 and Q2. Template matching. At inference, yes, and the revised Section 2 says so before the theory: the read-out is a matched filter on class averages, and the label amplitude plays no role in it. What the Hopfield-RBM view adds is on the training side, which is what Q2 asks. It fixes the scale N\sqrt N and the amplitude cLc_L at which a set of labels has to be clamped on the hidden layer for supervised CD-1 to leave the class averages nearly in place. On measured chords this amplitude leaves the smallest drift among the clamps we tested: at L=3L=3, 0.1710.171 at gLg_L, 0.2790.279 at cLc_L and 0.7790.779 at the binary clamp (Appendix G.3).

W2. "Theorem 1 holds by construction." We respectfully see this differently, and we have expanded Remark 1 so that the non-trivial step is clear in the text. The amplitude is chosen so that the positive phase matches the response of the class averages to a clean mixture. The content of the theorem is that the negative phase actually returns that response when it is computed by one step of Gibbs sampling at finite β\beta, from class averages of MM noisy examples, with a cue voted from BB items, at finite NN. Each of these moves the response away from the clean one, and Theorem 1 bounds each of them separately, in four channels that vanish at different rates. With a unit clamp the same computation leaves a difference of (1−cL)N(1-c_L)\sqrt N that none of the limits removes. Remark 1 now says both things in the paper. Lemma 1 is classical and we cite it as such. Lemma 2 is the classical arcsine identity applied to the encoder.

W2. The theory on real data. We agree, and the revised conclusion says it. The guarantees are proved for Rademacher archetypes with site-independent corruption. The bound is vacuous at the experimental parameters, because its dataset channel is of order one at the noise levels of measured data (Appendix G.4), and CD-1 moves the weights off the class averages. The effective number of independent sites, far below NN, is measured in Appendix D and makes the bound worse still. On real data the theorem is a hypothesis we test, and Section 4 reports where it fails.

W2. Absolute performance. We agree that it is modest. On Samson at L=2L=2 the comparison with "predict every class" is dominated by the metric: with two of three classes present, returning all three already gives micro-F1 0.8000.800, and under the parameter-free rule every filter in Table 5 falls below it. Told the order, our read-out reaches 0.5670.567 there, sparse recovery 0.7790.779 and least squares 0.7150.715, for the reason given under Q1. OpenMIC, with 0.5340.534 macro-AP against 0.4470.447 chance, is the setting farthest from the theory: there are no isolated sources, so the prototypes are averages of mixtures (Appendix D.6). We report these numbers because they mark the limits of the construction, not to claim competitive performance.

I thank the authors for their detailed rebuttal and additional experiments. However, my main concern remains unresolved.

The practical value of the proposed Hopfield–RBM framework is still not convincingly demonstrated. The authors acknowledge that inference reduces to a simple matched filter, while the new results in Table 2 show that plain class-average initialisation achieves essentially the same performance as the proposed method. Furthermore, the theoretical guarantees do not apply meaningfully to the real datasets, where the bound is vacuous and training moves the weights away from the theoretically stationary configuration.

I therefore maintain my original rating of 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

What changed in the revised manuscript

We are grateful to the reviewers for their thorough assessments and concrete suggestions, which substantially improved the clarity of our claims and the breadth of our experiments. We have uploaded a revised PDF with every change in red. Here we list what changed and say plainly what the paper claims. The replies under each review go through the points one by one.

What changed

  1. We no longer claim an exact fixed point in mean. Definition 1 now defines δ\delta-stationarity in mean, ∣E[Δwiμ]∣/(ϵβ)≤δ|\mathbb E[\Delta w_i^\mu]|/(\epsilon\beta)\le\delta for every entry, and calls a configuration asymptotically stationary when δ\delta can be made to vanish. Theorem 1 keeps its four-channel bound and now ends with the limit: the normalised drift vanishes as β\beta, MM, BB and NN grow, with ln⁡(KT)=o(N)\ln(KT)=o(N), in any order, and the update itself vanishes as long as ϵβ\epsilon\beta stays bounded. The abstract, the contributions and the conclusion have been changed accordingly.

  2. A new paragraph in Section 2, just before Definition 1, separates the three things the construction does: it gives the weights (the class averages), it reads a mixture with them (a matched filter, in which the label amplitude plays no role), and it trains them with supervised CD-1, which is where the amplitude enters.

  3. The introduction now discusses the compositional regime of Tubiana & Monasson (2017) and presents our setting as its supervised counterpart. Remark 2 says what Theorem 1 adds to Leonelli et al. (2021) and to a related bound for coupled RBMs. Pham et al. (2026) is cited, and the opening no longer says that spurious states have only been treated as defects.

  4. Remark 1 now says that the amplitude is a choice, and that it is Theorem 1 that shows the sampled negative phase returns it. It also explains why a binary clamp fails: it leaves a gap of (1−cL)N(1-c_L)\sqrt N between the two phases that no limit closes. Section 4.1 explains why the score cannot tell the amplitudes apart while the measured drift can.

  5. The abstract now says what CD-1 does there: it moves the weights off the class averages, rebalancing the classes and decorrelating the archetypes. The conclusion adds that the guarantees are proved for Rademacher archetypes with site-independent corruption, and that on measured signals the bound is vacuous, because its dataset channel is of order one at the noise levels of real data.

  6. The abstract no longer calls the initialisation "optimal", and the steps saved are stated relative to a random initialisation of the same scale. The introduction no longer speaks of an exact characterization of the dynamics: the result is a bound on the mean of one update. The read-out that is compared with one-bit compressed sensing is no longer called parameter-free, since that comparison is made with the number of sources given. Section 4.2 and Appendix F say that Tables 4 and 5 evaluate the weights at step zero, before any CD-1 update.

  7. Three new experiments, all in the appendices:

    • modern Hopfield read-outs (softmax, sparsemax, 1.5-entmax) when the number of labels is unknown (Appendix F.1, Table 6);
    • supervised CD-1 from nine starting points (Appendix E, Table 2);
    • non-negative least squares on the real-valued signals (Appendix F.2, Table 7).
AuthorsOpenReview ↗

Claims of the paper

At inference, the network acts as a matched filter: each hidden unit correlates the query with the class average of its archetype, and a parameter-free threshold selects the active constituents. Because the read-out operates on one-bit (signed) codes, it is not designed to beat full-precision templates; accordingly, at step zero under the parameter-free rule, the full-precision archetype filter (the nearest-centroid classifier built on pre-sign prototypes) achieves higher micro-F1 across all cells of Table 5. When the order LL is given, however, the class average of the codes outperforms the archetype filter on the deepest Urban mixtures (0.9150.915 vs. 0.8750.875 in micro-F1 and 0.5730.573 vs. 0.3770.377 in exact match at L=5L=5, Table 4). Against iterative one-bit sparse recovery baselines (BIHT, OMP, LASSO) on the same codes and with LL given, it matches or outperforms them in three of the four settings while requiring 6262 to 557557 times less arithmetic (Table 3).

The main claim is about training. The Hopfield–RBM duality fixes both the weight scale N\sqrt{N} and the clamping amplitude cLc_L required on the hidden layer for class averages to remain approximately stationary under supervised CD-1, with Theorem 1 bounding the residual drift. On measured chords, the arcsine amplitude gLg_L yields the smallest drift among the tested clamps: at L=3L=3, the drift is 0.1710.171 for gLg_L, compared to 0.2790.279 for cLc_L and 0.7790.779 for a binary clamp (Appendix G.3). Across the four benchmarks, class averages start 0.080.08 to 0.430.43 above random weights of the same scale—a gap that takes 4⋅1024\cdot10^2 to 1.6⋅1031.6\cdot10^3 CD-1 steps to close in three settings and is never closed in the fourth. When archetypes overlap (Urban, Samson), subsequent CD-1 updates improve upon the class averages themselves by +0.038+0.038 and +0.157+0.157 in micro-F1 (Figure 2) by rebalancing classes and decorrelating the archetypes. Table 2 shows how this initialization advantage depends on the baseline initialization, which we discuss further in our responses to Lxj8 and Bkxq.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.