- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an energy-based recurrent state-space model for action-conditioned visual prediction by combining hierarchical energy-based modeling, predictive coding, and continuous attractor neural networks (CANNs). The model generates latent states from a memory-conditioned prior, predicts observations through a hierarchical generative network, and updates its parameters and memory through local learning rules and sampling-based inference. The authors further introduce a prediction-before-inference learning objective and evaluate the framework on several visual prediction tasks, including eye movements, navigation-like environments, head turning, MNIST-rot, and TaxiBJ.
Critical strengths
The paper addresses action-conditioned visual prediction and attempts to integrate hierarchical generative modeling with recurrent memory. This is a reasonable research direction, and the proposed framework provides a unified perspective on prediction, inference, and memory updating.
Critical weaknesses
1、The paper motivates its approach by arguing that existing world models often rely on backpropagation through time (BPTT) and variational inference, which are not fully aligned with biologically plausible neural computation. However, the manuscript does not sufficiently explain why this gap constitutes a significant limitation for the predictive-learning problems considered, or why the proposed combination of hierarchical energy-based modeling and continuous attractor neural networks is particularly well suited to address it. 2、The proposed framework combines several established research directions, including hierarchical energy-based models, predictive coding, continuous attractor neural networks, and recurrent state-space models. Such integration can be valuable, but the manuscript does not sufficiently establish which specific technical contribution goes beyond the combination of these existing components.
Questions for authors
1、The paper motivates its approach by the gap between conventional world models and biologically plausible neural computation. What specific limitation of existing predictive-learning or world-model approaches does the proposed framework address? Why are hierarchical energy-based modeling and continuous attractor neural networks particularly suitable for addressing this limitation, beyond providing an alternative biologically inspired architecture? 2、The proposed framework combines hierarchical energy-based models, predictive coding, CANNs, and recurrent state-space modeling. What is the principal technical contribution that distinguishes this framework from closely related approaches? In particular, what is fundamentally new about the prediction-before-inference objective, and how does it differ from existing predictive-learning or energy-based learning objectives in terms of optimization mechanism and learning behavior? 3、The experiments primarily evaluate predictive performance using MSE, whereas the motivation emphasizes biologically plausible learning and structured attractor memory. Which experimental results directly demonstrate that these design choices provide meaningful benefits? In particular, how much of the observed performance can be attributed to the proposed learning objective and the CANN memory mechanism, respectively, rather than to the overall model architecture? 4、Equations (5)–(7) define a prediction objective based on samples from the memory-conditioned prior, but the presented gradient is with respect to the energy-model parameters. Which parameters of the prior distribution and memory dynamics are actually optimized, and through what mechanism? How does the implemented optimization support the claim that the proposed objective directly improves the memory-conditioned prior? 5、The paper claims generalization to unseen environments, but the described motion and head-turning experiments use training-phase sequences to initialize memory. How are the training and testing environments separated, and which results specifically demonstrate generalization to environments that were not encountered during training?
3 comments
3. What do the experiments establish about the design choices?
The existing results establish predictive capability of the complete system:
| Claim supported by the submitted evidence | Evidence and location |
|---|---|
| Generation from partial observations | Figure 3(a,b,d), with clearer examples in Figure 5(b) |
| Action-conditioned visual prediction | Figure 4(a) |
| Dependence on model capacity and observation count | Figure 4(b–e) |
| Temporal generation after initialization | Figure 6(a,e) |
| Quantitative comparisons with other predictors | Tables 1–3 |
For example, Table 1 reports MSE 0.0770 versus 0.1179 for TransDreamer at , a 34.7% reduction, and 0.0287 versus 0.0471 at , a 39.1% reduction. Our model has lower MSE in all reported Table 1 settings. In the unseen columns of Table 2, the row labeled 1024 reports 0.0087 versus 0.0151 for tPC.
We also avoid overstating the comparison: in Table 3, our fourth-frame TaxiBJ error is 0.633, compared with 0.762 for ST-ResNet, 0.721 for VPN, and 0.619 for FRNN. This is competitive performance, not a win over every baseline.
None of these comparisons isolates the causal contribution of the learning schedule or proves that the memory implements a continuous attractor. Figure 3/4 capacity studies are architectural sensitivity studies, not substitutes for a learning-order ablation or a matched reservoir control. Section 6’s qualitative statement about posterior-based learning should not be treated as quantitative evidence. We will remove that superiority claim unless a controlled numerical comparison is supplied.
Our position is therefore specific: MSE is appropriate for assessing the stated visual-prediction task, but insufficient for establishing a biological mechanism or attractor geometry. The latter claims require different evidence.
4. Which prior and memory parameters are actually optimized?
The learned parameters are the hierarchical generative weights in Eq. (16), updated locally as in Eq. (21). The implementation uses fixed precision matrices, and Appendix C specifies a fixed recurrent matrix . There is no learned recurrent-transition parameter update in Algorithm 1.
The top-level prior is Eq. (17),
At fixed current memory, updating the decoder does not directly optimize this prior’s mean or precision. Learned weights affect subsequent inferred latent states; those states drive subsequent memory updates; the resulting memory changes future conditional predictions. This explains how the effective predictor can adapt without implying that gradients are propagated through that dependency chain. Lower-level generative conditionals are parameterized by the learned inter-layer weights, which should not be confused with learning the top-level memory dynamics.
Writing the per-step criterion as , the implemented update retains the fixed-memory partial derivative and omits
Thus, Eq. (7) should be interpreted with the sampling distribution/current memory held fixed, and the implemented layerwise procedure is a local/semi-gradient approximation rather than exact full-sequence optimization. Section 6 already acknowledges the omitted memory dependency, but the main text should state it where the gradient is introduced.
We will replace “directly improves/trains the memory-conditioned prior” with “trains predictions generated from memory-conditioned prior states.” This correction preserves the actual algorithm while removing an unsupported claim about the optimization pathway.
5. What generalizes, and what does not?
The reviewer is correct that the motion/head-turning experiments do not establish generalization to unseen environments. Section 5 describes initializing memory with training-phase sequences and predicting within the learned environment. Figure 4 should be interpreted as within-environment action-conditioned generation, not transfer to a held-out map or street.
There are distinct existing tests of generalization:
- Figures 3(a) and 5(b) explicitly include images not used to train the corresponding model, supporting unseen-image generation after partial-observation initialization without weight updates.
- Section 5 and the unseen columns of Table 2 report evaluation on unseen temporal sequences.
- Figure 6(e) and Table 3 report the TaxiBJ temporal-prediction setting described in Section 5; this is a sequence-level setting, not evidence of transfer to a new geographical environment.
We will therefore replace the broad “unseen environments” wording with the narrower supported description: “generalization to unseen images and temporal sequences, together with action-conditioned generation within learned environments.” We will not retrospectively label a training-sequence initialization protocol as a held-out-environment split.
Thank you for the response. I am keeping my rating at 2. I still find it difficult to assess what training on the prior state contributes to those predictions, and what the attractor-inspired memory contributes when the agent changes its action.
Response to Reviewer dcfn
We thank the reviewer for identifying the central questions: the problem addressed, the technical contribution beyond component integration, causal evidence for the design choices, the actual optimization path, and the scope of generalization. Our main claim is not that adding biological terminology improves an otherwise standard predictor. It is that an explicit action-conditioned predictive-learning loop can be implemented using local generative-weight updates and neural-state inference, without propagating gradients through recurrent memory dynamics. We agree that the manuscript needs sharper boundaries between this demonstrated procedure and stronger claims about attractors, prior learning, or transfer.
1. What limitation does the framework address?
The limitation is a learning-mechanism constraint, not an assertion that BPTT prevents conventional world models from predicting accurately. When learning signals cannot be propagated through an unfolded history, the model must connect prediction, state inference, and recurrent memory by a different mechanism.
The proposed construction assigns each component a specific role:
Energy-based modeling is useful here because it expresses state inference as dynamics and, in the Gaussian implementation, yields a local parameter-update rule. Recurrent memory is useful because the instantaneous observation is generally insufficient for predicting the next observation after an action.
A continuous attractor, however, has not been shown to be necessary for this procedure, nor has its existence been established for the random low-rank implementation. We will distinguish the attractor-inspired motivation from the demonstrated recurrent computation. We also do not claim that the model is a validated description of biological circuitry; local updates alone would not establish that stronger conclusion.
2. What is technically new, and what is not?
We do not claim to introduce energy-based models, predictive coding, RSSMs, Hebbian plasticity, Jensen’s inequality, or prior-predictive training in isolation. The technical contribution is the specific layerwise learning/inference schedule and its integration into a recurrent predictive system:
This distinction is explicit in Eqs. (9)–(11) and Algorithm 1. In the hierarchy, the procedure is applied layerwise: Eqs. (20)–(24) use the lower-layer observation/inferred state as a target while retaining a prior-generated predictor at the layer being updated. It is therefore more precise than describing all layers as undergoing a single simultaneous global update.
The fixed-memory objective is
L=−EoEp(s∣m)logpθ(o∣s),
and Eq. (33) gives
L=H+EoDKL[p(s∣m)∥pθ(s∣o,m)].
This is not a fundamentally separate class of probabilistic bounds: at a fixed memory, it can also be written as the negative ELBO obtained by choosing the variational distribution equal to the prior. The substantive distinction is the sampling/update mechanism and its local recurrent implementation, not a new variational identity. Conversely, Eq. (8) is a posterior reconstruction term, not a complete characterization of modern variational world-model objectives, which can also train prediction through additional terms. We will make that distinction explicit rather than imply that existing world models only reconstruct.
The claim we defend is thus a concrete implementable learning procedure—not a claim that the underlying probabilistic ingredients are individually unprecedented.