acceptodds
Under review as a conference paper at ICLR 2027

Predictive Learning in Energy-based Models with Attractor Structures

Abstract

Predictive models are highly advanced in understanding the mechanisms of brain function. Recent advances in machine learning further underscore the power of prediction for optimal representation in learning. However, there remains a gap in creating a biologically plausible model that explains how the neural system achieves prediction. In this paper, we introduce a framework that employs an energy-based model (EBM) to capture the nuanced processes of predicting observation after action within the neural system, encompassing prediction, learning, and inference. We implement the EBM with a hierarchical structure and integrate a continuous attractor neural network for memory, constructing a biologically plausible model. In experimental evaluations, our model demonstrates efficacy across diverse scenarios. The range of actions includes eye movement, motion in environments, head turning, and static observation while the environment changes. Our model not only makes accurate predictions for environments it was trained on, but also provides reasonable predictions for unseen environments, matching the performances of machine learning methods in multiple tasks. We hope that this study contributes to a deep understanding of how the neural system performs prediction.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-inspired AI, World models, Neural Sampling, Energy-based models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dcfnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an energy-based recurrent state-space model for action-conditioned visual prediction by combining hierarchical energy-based modeling, predictive coding, and continuous attractor neural networks (CANNs). The model generates latent states from a memory-conditioned prior, predicts observations through a hierarchical generative network, and updates its parameters and memory through local learning rules and sampling-based inference. The authors further introduce a prediction-before-inference learning objective and evaluate the framework on several visual prediction tasks, including eye movements, navigation-like environments, head turning, MNIST-rot, and TaxiBJ.

Critical strengths

The paper addresses action-conditioned visual prediction and attempts to integrate hierarchical generative modeling with recurrent memory. This is a reasonable research direction, and the proposed framework provides a unified perspective on prediction, inference, and memory updating.

Critical weaknesses

1、The paper motivates its approach by arguing that existing world models often rely on backpropagation through time (BPTT) and variational inference, which are not fully aligned with biologically plausible neural computation. However, the manuscript does not sufficiently explain why this gap constitutes a significant limitation for the predictive-learning problems considered, or why the proposed combination of hierarchical energy-based modeling and continuous attractor neural networks is particularly well suited to address it. 2、The proposed framework combines several established research directions, including hierarchical energy-based models, predictive coding, continuous attractor neural networks, and recurrent state-space models. Such integration can be valuable, but the manuscript does not sufficiently establish which specific technical contribution goes beyond the combination of these existing components.

Questions for authors

1、The paper motivates its approach by the gap between conventional world models and biologically plausible neural computation. What specific limitation of existing predictive-learning or world-model approaches does the proposed framework address? Why are hierarchical energy-based modeling and continuous attractor neural networks particularly suitable for addressing this limitation, beyond providing an alternative biologically inspired architecture? 2、The proposed framework combines hierarchical energy-based models, predictive coding, CANNs, and recurrent state-space modeling. What is the principal technical contribution that distinguishes this framework from closely related approaches? In particular, what is fundamentally new about the prediction-before-inference objective, and how does it differ from existing predictive-learning or energy-based learning objectives in terms of optimization mechanism and learning behavior? 3、The experiments primarily evaluate predictive performance using MSE, whereas the motivation emphasizes biologically plausible learning and structured attractor memory. Which experimental results directly demonstrate that these design choices provide meaningful benefits? In particular, how much of the observed performance can be attributed to the proposed learning objective and the CANN memory mechanism, respectively, rather than to the overall model architecture? 4、Equations (5)–(7) define a prediction objective based on samples from the memory-conditioned prior, but the presented gradient is with respect to the energy-model parameters. Which parameters of the prior distribution and memory dynamics are actually optimized, and through what mechanism? How does the implemented optimization support the claim that the proposed objective directly improves the memory-conditioned prior? 5、The paper claims generalization to unseen environments, but the described motion and head-turning experiments use training-phase sequences to initialize memory. How are the training and testing environments separated, and which results specifically demonstrate generalization to environments that were not encountered during training?

3 comments

Response to Reviewer dcfn

We thank the reviewer for identifying the central questions: the problem addressed, the technical contribution beyond component integration, causal evidence for the design choices, the actual optimization path, and the scope of generalization. Our main claim is not that adding biological terminology improves an otherwise standard predictor. It is that an explicit action-conditioned predictive-learning loop can be implemented using local generative-weight updates and neural-state inference, without propagating gradients through recurrent memory dynamics. We agree that the manuscript needs sharper boundaries between this demonstrated procedure and stronger claims about attractors, prior learning, or transfer.

1. What limitation does the framework address?

The limitation is a learning-mechanism constraint, not an assertion that BPTT prevents conventional world models from predicting accurately. When learning signals cannot be propagated through an unfolded history, the model must connect prediction, state inference, and recurrent memory by a different mechanism.

The proposed construction assigns each component a specific role:

  • The hierarchical generative model, Eqs. (15)–(16), connects high-dimensional observations to latent states. Fixed-precision Gaussian conditionals yield local prediction-error updates such as Eq. (21).
  • Sampling-based inference, Eqs. (12)–(13), integrates the new observation with the memory-conditioned prediction without requiring an amortized posterior encoder trained through the recurrent history.
  • Recurrent memory, Eqs. (17), (41)–(43), carries history and action information forward between observations.
  • Algorithm 1 specifies the resulting executable prediction–learning–inference–memory cycle.

Energy-based modeling is useful here because it expresses state inference as dynamics and, in the Gaussian implementation, yields a local parameter-update rule. Recurrent memory is useful because the instantaneous observation is generally insufficient for predicting the next observation after an action.

A continuous attractor, however, has not been shown to be necessary for this procedure, nor has its existence been established for the random low-rank implementation. We will distinguish the attractor-inspired motivation from the demonstrated recurrent computation. We also do not claim that the model is a validated description of biological circuitry; local updates alone would not establish that stronger conclusion.

2. What is technically new, and what is not?

We do not claim to introduce energy-based models, predictive coding, RSSMs, Hebbian plasticity, Jensen’s inequality, or prior-predictive training in isolation. The technical contribution is the specific layerwise learning/inference schedule and its integration into a recurrent predictive system:

  1. Generate predictions from memory before incorporating the current observation.
  2. When the observation arrives, update the relevant generative weights using the retained prior-prediction state.
  3. Infer the observation-conditioned latent state.
  4. Use the inferred state and the next action to update recurrent memory.

This distinction is explicit in Eqs. (9)–(11) and Algorithm 1. In the hierarchy, the procedure is applied layerwise: Eqs. (20)–(24) use the lower-layer observation/inferred state as a target while retaining a prior-generated predictor at the layer being updated. It is therefore more precise than describing all layers as undergoing a single simultaneous global update.

The fixed-memory objective is

L=−EoEp(s∣m)log⁡pθ(o∣s),\mathcal L=-\mathbb E_o\mathbb E_{p(s\mid m)}\log p_\theta(o\mid s),

and Eq. (33) gives

L=H+EoDKL[p(s∣m) ∥ pθ(s∣o,m)].\mathcal L=\mathcal H+\mathbb E_oD_{\mathrm{KL}}\left[p(s\mid m)\,\|\,p_\theta(s\mid o,m)\right].

This is not a fundamentally separate class of probabilistic bounds: at a fixed memory, it can also be written as the negative ELBO obtained by choosing the variational distribution equal to the prior. The substantive distinction is the sampling/update mechanism and its local recurrent implementation, not a new variational identity. Conversely, Eq. (8) is a posterior reconstruction term, not a complete characterization of modern variational world-model objectives, which can also train prediction through additional terms. We will make that distinction explicit rather than imply that existing world models only reconstruct.

The claim we defend is thus a concrete implementable learning procedure—not a claim that the underlying probabilistic ingredients are individually unprecedented.

3. What do the experiments establish about the design choices?

The existing results establish predictive capability of the complete system:

Claim supported by the submitted evidenceEvidence and location
Generation from partial observationsFigure 3(a,b,d), with clearer examples in Figure 5(b)
Action-conditioned visual predictionFigure 4(a)
Dependence on model capacity and observation countFigure 4(b–e)
Temporal generation after initializationFigure 6(a,e)
Quantitative comparisons with other predictorsTables 1–3

For example, Table 1 reports MSE 0.0770 versus 0.1179 for TransDreamer at N=128,K=4N=128,K=4, a 34.7% reduction, and 0.0287 versus 0.0471 at N=128,K=16N=128,K=16, a 39.1% reduction. Our model has lower MSE in all reported Table 1 settings. In the unseen columns of Table 2, the row labeled 1024 reports 0.0087 versus 0.0151 for tPC.

We also avoid overstating the comparison: in Table 3, our fourth-frame TaxiBJ error is 0.633, compared with 0.762 for ST-ResNet, 0.721 for VPN, and 0.619 for FRNN. This is competitive performance, not a win over every baseline.

None of these comparisons isolates the causal contribution of the learning schedule or proves that the memory implements a continuous attractor. Figure 3/4 capacity studies are architectural sensitivity studies, not substitutes for a learning-order ablation or a matched reservoir control. Section 6’s qualitative statement about posterior-based learning should not be treated as quantitative evidence. We will remove that superiority claim unless a controlled numerical comparison is supplied.

Our position is therefore specific: MSE is appropriate for assessing the stated visual-prediction task, but insufficient for establishing a biological mechanism or attractor geometry. The latter claims require different evidence.

4. Which prior and memory parameters are actually optimized?

The learned parameters are the hierarchical generative weights θl\theta^l in Eq. (16), updated locally as in Eq. (21). The implementation uses fixed precision matrices, and Appendix C specifies a fixed recurrent matrix WW. There is no learned recurrent-transition parameter update in Algorithm 1.

The top-level prior is Eq. (17),

p(stL∣mt)=N(mt,(ΛL)−1).p(s_t^L\mid m_t)=\mathcal N(m_t,(\Lambda^L)^{-1}).

At fixed current memory, updating the decoder does not directly optimize this prior’s mean or precision. Learned weights affect subsequent inferred latent states; those states drive subsequent memory updates; the resulting memory changes future conditional predictions. This explains how the effective predictor can adapt without implying that gradients are propagated through that dependency chain. Lower-level generative conditionals are parameterized by the learned inter-layer weights, which should not be confused with learning the top-level memory dynamics.

Writing the per-step criterion as Lt(θ,mt)\mathcal L_t(\theta,m_t), the implemented update retains the fixed-memory partial derivative and omits

∂Lt∂mtdmtdθ.\frac{\partial\mathcal L_t}{\partial m_t}\frac{dm_t}{d\theta}.

Thus, Eq. (7) should be interpreted with the sampling distribution/current memory held fixed, and the implemented layerwise procedure is a local/semi-gradient approximation rather than exact full-sequence optimization. Section 6 already acknowledges the omitted memory dependency, but the main text should state it where the gradient is introduced.

We will replace “directly improves/trains the memory-conditioned prior” with “trains predictions generated from memory-conditioned prior states.” This correction preserves the actual algorithm while removing an unsupported claim about the optimization pathway.

5. What generalizes, and what does not?

The reviewer is correct that the motion/head-turning experiments do not establish generalization to unseen environments. Section 5 describes initializing memory with training-phase sequences and predicting within the learned environment. Figure 4 should be interpreted as within-environment action-conditioned generation, not transfer to a held-out map or street.

There are distinct existing tests of generalization:

  • Figures 3(a) and 5(b) explicitly include images not used to train the corresponding model, supporting unseen-image generation after partial-observation initialization without weight updates.
  • Section 5 and the unseen columns of Table 2 report evaluation on unseen temporal sequences.
  • Figure 6(e) and Table 3 report the TaxiBJ temporal-prediction setting described in Section 5; this is a sequence-level setting, not evidence of transfer to a new geographical environment.

We will therefore replace the broad “unseen environments” wording with the narrower supported description: “generalization to unseen images and temporal sequences, together with action-conditioned generation within learned environments.” We will not retrospectively label a training-sequence initialization protocol as a held-out-environment split.

Thank you for the response. I am keeping my rating at 2. I still find it difficult to assess what training on the prior state contributes to those predictions, and what the attractor-inspired memory contributes when the agent changes its action.

Reviewer_j9d3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a brain-inspired world model that leverages predictive coding based on energy minimization, a continuous attractor based memory network, and hebbian updates. Inference is done by langevin energy minimization.

Critical strengths
  • The best strength about this paper is that many of the ideas are biologically inspired, and so the framework is much more brain-like than many current world model systems, involving hebbian updates and an explicit neural memory.
  • The proposed framework, if very scalable empirically, could likely be mapped onto very fast analog hardware and thus scaled well from a systems perspective.
  • The proposed approach and results are genuinely interesting and novel.
Critical weaknesses
  • The biggest weakness is that many of the results are on very toy tasks, and so we don't know if the proposed approach scales well. There have been several proposed approaches that appear to be biologically plausible, but almost none of them have scaled very well. This remains the major challenge for more brain-inspired algorithms, and appears to be a core challenge for the proposed approach in this work, with modest results on mnist and cifar. If the authors were able to address this and scale to much harder benchmarks, such as atari or even deepmind control suite, that would be significantly more convincing for the approach. Or, the authors could compare the current approach to a FLOP matched BPTT baseline to further demonstrate its benefits at small scale, if scaling to larger scale remains infeasible.
  • The CANN weights are not learned, so that limits scalability to very high dimensional real world tasks
Questions for authors

Can we do some more controlled ablations on different design choices? I.e., how does langevin compare to other samplers, how does hebbian learning compare to BPTT or local learning approaches that are decently popular (FA, PC, etc)? Can we learn the neural memory?

3 comments

Response to Reviewer j9d3

We thank the reviewer for recognizing the novelty and interest of the framework, its local learning rules, and its potential compatibility with alternative computing substrates. We agree that scalability is a central challenge for biologically inspired learning. Our position is that the paper should establish what can be achieved under local-learning and no-temporal-backpropagation constraints, while clearly separating this achievement from an unproven claim of large-scale superiority. We address the scale, compute, ablation, and memory questions below.

1. The evaluation is not confined to MNIST and CIFAR, although its scale remains limited

We agree that the image-patch experiments are controlled small-scale tasks. However, the submission also includes navigation-like prediction, street-view motion/head turning, and real traffic-flow prediction:

  • DeepMind Lab: Figure 4(a–c) reports visual prediction under movement actions. Table 4 specifies observations of 3×80×603\times80\times60, or 14,400 scalar inputs per observation.
  • Street-view motion/head turning: Figure 4(a,d,e) reports action-conditioned visual predictions with 3×100×503\times100\times50, or 15,000 scalar inputs per observation, as specified in Table 4.
  • TaxiBJ: Figure 6(e,f) and Table 3 evaluate traffic-flow sequences. The comparison in Table 3 uses four historical frames to predict four subsequent frames.

These experiments show that the implemented learning procedure is not restricted to handwritten digits or a static-image reconstruction setting. Input dimensionality is nevertheless not equivalent to dataset scale, environmental diversity, or control competence. The navigation and street-view results concern learned environments; they do not demonstrate large-scale transfer or Atari/DMControl performance.

The existing quantitative evidence is also stronger than a qualitative demonstration alone:

Existing resultOurs, MSE ↓Comparator, MSE ↓Location
Eye movement, N=128,K=4N=128,K=40.0770TransDreamer: 0.1179Table 1
Eye movement, N=128,K=8N=128,K=80.0606TransDreamer: 0.0911Table 1
Eye movement, N=128,K=16N=128,K=160.0287TransDreamer: 0.0471Table 1
MNIST-rot, unseen, row labeled 10240.0087tPC: 0.0151Table 2
TaxiBJ, future frame 30.567ST-ResNet: 0.670; VPN: 0.645; FRNN: 0.518Table 3
TaxiBJ, future frame 40.633ST-ResNet: 0.762; VPN: 0.721; FRNN: 0.619Table 3

Table 1 favors our model in all 12 reported settings. Table 3, importantly, does not show universal superiority: FRNN is better on all four frames. Our interpretation is competitive predictive behavior under a different learning constraint, not state-of-the-art forecasting.

We therefore agree to narrow the scalability language, but respectfully disagree with treating the empirical contribution as only modest MNIST/CIFAR reconstruction. The paper also demonstrates action-conditioned generation and non-digit temporal prediction under the proposed learning procedure.

2. Parameter matching is not FLOP matching

The reviewer’s suggestion of a FLOP-matched BPTT baseline is well motivated. Section 5 and Table 1 describe a parameter-matched TransDreamer comparison; they do not establish equal training compute. Our dynamics require iterative inference, so the absence of BPTT does not automatically imply fewer operations or lower wall-clock time.

Appendix E and Table 4 report the numerical integration settings and hardware. For example, settings with T=10T=10 and dt=0.05dt=0.05 entail 200 numerical integration steps over a simulated phase. This cost must be included in a fair compute comparison, along with layerwise inference, learning, and memory updates. Appendix E also reports approximately 10 hours for the best-performing navigation/street-view configurations on an RTX A6000.

Accordingly, we will not claim a demonstrated computational speedup or hardware advantage. The established property is the absence of gradient propagation through recurrent memory dynamics, not a measured FLOP reduction. Analog-hardware efficiency remains a prospective opportunity, not an experimental result of this paper.

3. Existing design studies versus the requested controlled ablations

The submission already contains several design studies:

  • Hierarchy depth: Figure 3(c) and Figure 5(c) compare different depths under a fixed total latent-neuron count. This controls neuron count, not necessarily parameter count or FLOPs.
  • Initialization information: Figure 3(d) varies the number of observed patches; Figure 3(e) compares memory initialization with K=8K=8 and K=16K=16 as model size changes.
  • Capacity and environmental observations: Figure 4(b–e) varies network size and the number of observations to be represented. These plots expose capacity limitations rather than hide them.
  • Temporal-model capacity: Figure 6(b,c,f) evaluates temporal prediction as configuration size varies.

These results provide useful architectural sensitivity evidence. They do not substitute for Langevin-versus-other-sampler, Hebbian-versus-BPTT, or feedback-alignment comparisons, and we will label them accordingly.

The mechanisms are nevertheless explicitly specified: Eq. (13) gives sampling-based posterior inference; Eq. (21) gives the local prediction-error weight update; Algorithm 1 specifies how prediction, learning, inference, and memory interact. The present contribution is demonstrating this complete procedure, not establishing that each component is individually optimal.

4. Can the memory be learned?

Yes in principle, but the current implementation does not learn the recurrent matrix WW. Appendix C specifies fixed randomly generated low-rank connectivity. What is learned is the hierarchical generative mapping, while latent states and recurrent activities change dynamically.

We agree that this is a plausible capacity bottleneck. Figure 4(c,d) directly shows performance degradation as the number of environmental observations increases at a fixed model size, while the larger-network points show improved capacity. These observations motivate trainable memory, but they do not establish that training WW alone will solve scaling.

It is also important to distinguish two possible extensions. Training the memory with BPTT would produce a useful engineering variant but relax our learning constraint; training it with an explicit local plasticity rule would preserve that constraint but requires a separately specified and evaluated mechanism. We will not describe either extension as already implemented. Likewise, because random low-rank recurrence alone does not establish a continuous attractor, we will qualify the current memory as attractor-inspired unless its dynamics are directly verified.

Closing perspective

Large-scale control and compute-matched comparisons remain open; the existing results establish functional feasibility, not scalability at arbitrary scale. At the same time, a framework that combines local predictive updates, recurrent state, action conditioning, and generation after partial observations is a meaningful result even before it reaches Atari-scale control. We respectfully ask that the paper be assessed on this concrete contribution and its already reported quantitative comparisons, without interpreting our clarification of scope as a claim that the larger-scale challenge has been solved.

Thanks for the detailed rebuttal, and fair point that the evaluation goes beyond MNIST/CIFAR.

However, none of the requested ablations or compute-matched baselines were added, so my scalability concerns remain, and the proposed changes narrow the claims.

I will maintain my score, but I believe that the direction is promising.

Reviewer_NHA1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a biologically inspired predictive world model that combines a hierarchical energy-based network with a continuous attractor neural network (CANN) for recurrent memory. The model introduces a learning-before-inference objective, where synaptic parameters are updated using predictions sampled from the memory-conditioned prior, followed by posterior inference and memory updates. This framework aims to enable action-conditioned prediction and imagination through local learning rules without backpropagation through time (BPTT).

The authors evaluate the model on eye movement, navigation-like motion, head turning, and temporal prediction tasks. The results show competitive prediction performance against several recurrent and predictive-coding baselines.

Critical strengths
  1. An interesting biologically inspired predictive learning framework. The paper integrates energy-based learning, predictive coding, and recurrent memory into a unified framework. The learning-before-inference formulation is particularly interesting, as it emphasizes predictions generated from prior states rather than reconstruction from inferred posterior states.

  2. Experiments across different prediction scenarios. The model demonstrates promising results on several visual prediction tasks, including partial observations, action-conditioned prediction, and temporal sequences. Its competitive performance without BPTT is encouraging.

Critical weaknesses
  1. The continuous attractor property is not established. The recurrent memory uses a randomly generated low-rank connectivity matrix (Appendix C), but such a network does not necessarily form a continuous attractor manifold. The paper provides no analysis of attractor stability, manifold structure, or continuous attractor dynamics. As implemented, the memory module appears closer to a fixed random recurrent reservoir. Without comparisons against standard reservoir networks or other recurrent memory architectures, it is unclear whether the claimed advantages actually arise from continuous attractor dynamics. This is particularly concerning because attractor structures are a central contribution of the paper.

  2. Insufficient validation of the proposed learning objective. Learning-before-inference is presented as another key contribution, but the paper lacks a systematic ablation comparing it with inference-before-learning under the same architecture. Although the discussion mentions that posterior-based learning performs worse, no quantitative evidence is provided. Furthermore, the implementation ignores the dependence of memory on model parameters during gradient computation, making it unclear to what extent the proposed objective directly improves the memory-conditioned prior. These issues make it difficult to establish the benefits of the proposed learning strategy.

Questions for authors
  1. How is the continuous attractor property guaranteed or verified? Given the randomly initialized low-rank recurrent connectivity, can the authors provide evidence of a continuous attractor manifold and its stability? How does the model compare with a standard fixed random RNN or reservoir network?

  2. How much improvement comes specifically from learning-before-inference? Could the authors provide quantitative comparisons with posterior-based learning under identical architectures and training settings?

  3. How does ignoring gradients through the memory dynamics affect learning? Since the memory-conditioned prior depends on previous inferred states and learned parameters, could the authors clarify how the prior is improved without differentiating through these dependencies?

3 comments

Response to Reviewer NHA1

We thank the reviewer for recognizing the interest of prior-state predictive learning and the encouraging prediction results without temporal backpropagation. We agree that the paper must distinguish three separate claims: a functioning predictive-learning system, the benefit of a particular learning schedule, and the existence of a continuous attractor manifold. The submission provides evidence for the first, but does not yet establish the latter two to the same standard. Below, we clarify the implemented mechanism, identify the existing quantitative evidence, and specify which claims need to be narrowed.

1. Continuous attractors versus fixed recurrent memory

The reviewer is correct that random low-rank connectivity alone does not guarantee a continuous attractor. Appendix C, Eqs. (41)–(43), specifies a recurrent firing-rate system with fixed low-rank connectivity, sensory-state input, action input, and adaptation. Neither low rank nor stable-looking generated observations is sufficient to establish a continuum of fixed points with attraction transverse to that continuum.

Consequently, we will qualify the implementation as attractor-inspired low-rank recurrent memory, rather than present a verified continuous attractor manifold as an established result. This qualification must be reflected in the title, abstract, and discussion, not confined to a rebuttal footnote. We also accept that a reservoir interpretation is a legitimate alternative interpretation of the current memory module.

However, this limitation of the dynamical interpretation should not be conflated with an absence of predictive function. The submitted experiments already evaluate memory-conditioned generation:

  • Figure 3(a,b,d) shows generation after partial-observation initialization and the dependence on the number of initialization patches. Figure 5(b) provides clearer examples, including images not used for training.
  • Figure 4(a) shows action-conditioned predictions for virtual navigation and street-view motion/head turning. Figure 4(b–e) evaluates memory-system performance as network size and the number of environmental observations vary.
  • Figure 6(a,e) shows generated temporal sequences after initialization; Tables 2 and 3 report the corresponding predictive comparisons.

These results support the operation of the complete recurrent predictor. They are not a manifold-stability analysis or a comparison isolating attractor dynamics, and we will not present them as such. A matched reservoir comparison and perturbation/Jacobian analysis would be necessary to establish that stronger claim.

2. What is established about learning-before-inference?

The distinction is an operational one, not simply a relabeling of reconstruction. In Eqs. (9)–(11), the latent sample used for the update is drawn from the memory-conditioned prior before the current observation is incorporated. After the observation arrives, parameters are updated using this retained prediction; posterior inference then updates the state used for memory. Algorithm 1 makes this ordering explicit. Thus, “before inference” does not mean learning without observing the target.

For a fixed current memory, the criterion in Eq. (5) is

Lt=−EotEs^t∼p(st∣mt)log⁡pθ(ot∣s^t).\mathcal L_t=-\mathbb E_{o_t}\mathbb E_{\hat s_t\sim p(s_t\mid m_t)}\log p_\theta(o_t\mid\hat s_t).

The Jensen upper bound is valid. Appendix A, Eq. (33) further identifies its gap from predictive cross-entropy as an expected prior-to-posterior KL divergence. This gives a precise interpretation, but it does not by itself prove that the objective is empirically superior to posterior-state learning.

There is already quantitative evidence that the resulting learning system predicts effectively. For example, Table 1, for N=128N=128, reports:

Initialization patches KKOurs, MSE ↓TransDreamer, MSE ↓Relative reduction
40.07700.117934.7%
80.06060.091133.5%
160.02870.047139.1%

Our method has lower MSE in all 12 settings in that table. The unseen columns of Table 2 also favor our method over tPC in all reported rows; for example, the row labeled 1024 reports 0.0087 versus 0.0151.

These are whole-system comparisons, not the identical-architecture ablation requested by the reviewer. Section 6 mentions an inferior posterior-based variant, but does not provide its numerical results. We agree that this qualitative statement is insufficient to establish a causal advantage of the learning schedule. We will remove the unsupported comparative assertion unless it is accompanied by a reproducible quantitative comparison. Our defensible claim is that prior-state learning yields a workable local predictive-learning procedure—not that its superiority over posterior-state learning has already been isolated.

3. Which parameters are updated, and how can the prior change?

The implementation distinguishes three objects:

  1. Learned generative connections: the inter-layer weights θl\theta^l in Eq. (16), updated by local prediction errors, as illustrated by Eq. (21) and Algorithm 1.
  2. Fixed distributional and recurrent parameters: the precision matrices in the Gaussian implementation and the randomly constructed recurrent connectivity WW in Appendix C. The implementation does not learn WW.
  3. Dynamical states: inferred latent activities and memory activities, which change through inference and recurrent dynamics rather than parameter optimization.

The top-level prior is Eq. (17),

p(stL∣mt)=N(mt,(ΛL)−1).p(s_t^L\mid m_t)=\mathcal N(m_t,(\Lambda^L)^{-1}).

At a fixed mtm_t, its mean and precision are not directly trained by Eq. (7). Learning changes the decoder and the inferred states subsequently fed into memory, and therefore can change future memory states and future conditional predictions. This is indirect state-mediated adaptation, not a gradient update to the recurrent transition or the top-level prior parameters.

We therefore agree that “directly trains the memory-conditioned prior” is too strong for this implementation. We will replace it with: “trains predictions generated from memory-conditioned prior states using local generative-weight updates.”

4. What does ignoring memory gradients imply?

Writing the stepwise criterion as Lt(θ,mt)\mathcal L_t(\theta,m_t), the implemented update holds the current memory fixed. It omits the temporal dependency term in

= \left.\frac{\partial\mathcal L_t}{\partial\theta}\right|_{m_t} + \frac{\partial\mathcal L_t}{\partial m_t}\frac{dm_t}{d\theta}.$$ This is the approximation acknowledged in Section 6. It should be described as a local/semi-gradient update without backpropagation through recurrent memory dynamics, not as exact optimization of the full sequence objective. Longer predictions in Figures 4 and 6 demonstrate feasibility under this approximation, not the absence of a credit-assignment cost. In summary, we defend the paper’s concrete contribution: an explicit prior-prediction → local-learning → posterior-inference → memory-update procedure that works across several visual prediction settings. We accept the need to narrow the attractor and prior-optimization claims and to separate system-level performance from mechanism-specific evidence. We respectfully ask that the demonstrated predictive-learning procedure be assessed separately from the stronger dynamical interpretation that the present submission has not established.

Thank you for the clarifications and for appropriately narrowing the claims. While the framework demonstrates promising predictive performance, the distinction from existing reservoir-based approaches and the specific benefits of the proposed learning strategy remain insufficiently established. I therefore maintain my original assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.