- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SSP-JEPA, which replaces the learned latent slots in an object-centric world model with fixed vector-symbolic representations that bind an object's identity vector to a Spatial Semantic Pointer (SSP) encoding of its position. The authors evaluate the method on NRI springs and CLEVRER, with ground-truth object states as input, across three evaluation criteria (closed-loop rollout error, interaction-graph recovery, and counterfactual intervention), and report that it outperforms a coordinate-based GNN (coord-GNN) and a version of C-JEPA adapted to consume ground-truth object states.
Critical strengths
- The idea that a fixed SSP representation makes translation and relative position exact, closed-form algebraic operations, so that the predictor only needs to learn the interaction dynamics, is clear, and the method is described in sufficient detail to be reproduced.
- The experimental results are consistent. On NRI springs, the proposed method outperforms both coord-GNN and C-JEPA in closed-loop rollout, interaction-graph recovery, and counterfactual intervention. Graph recovery substantially exceeds the proximity baseline, which ranks edges purely by spatial distance, indicating that the model captures physical coupling rather than geometric co-location. On CLEVRER as well, the proposed method achieves the lowest error at contact frames, where constant-velocity extrapolation fails most severely.
Critical weaknesses
- The authors position this study as an object-centric world model and propose SSP-JEPA as a replacement for the slot representation of C-JEPA, using C-JEPA as the primary baseline. However, all experiments in this paper use ground-truth object states as input and do not address the central challenge in object-centric world model research: extracting object identities and positions from images. Under this setting, the readout without trained probes and the scene editing via algebraic operations, which this paper highlights as advantages over learned slots, do not constitute an advantage over coord-GNN, which uses raw coordinates directly as its state. Regarding C-JEPA as well, while the original method derives slots from video using SAVi or VideoSAUR, the version used here has been adapted to consume ground-truth object states with a coordinate displacement head; therefore, the comparison is against a Transformer predictor operating on coordinates, rather than an evaluation of C-JEPA as published. Although the authors acknowledge this limitation in Section 7, the temporally consistent discrete object identities and precise continuous coordinates required by the proposed representation are precisely the hardest part of extracting objects from images. Therefore, this study should be positioned as a study of representation and loss design for relational dynamics models that take ground-truth states as input, rather than as an object-centric world model; the effectiveness as an object-centric world model claimed in the current title, abstract, and conclusion is not supported by the experiments.
- There are numerous inconsistencies between the manuscript's descriptions and its claims, making it difficult to assess the reliability of the results. For example, the abstract reports 0.58× for C-JEPA and 0.68× for coord-GNN, whereas Table 1 and Section 5.1 report 0.65 and 0.64, respectively, so the reported values are inconsistent. Furthermore, while the introduction and Contribution 1 state "editing a scene by deletion, translation or insertion", only deletion is actually evaluated. Additionally, the main text contains several typographical errors, which further undermine the manuscript's credibility.
Questions for authors
- The difference from coord-GNN is the basis for the paper's central claim, but Table 2 shows that replacing the input-side components (edge features and history) with Cartesian coordinates causes only a small drop. In contrast, the "Metric displacement loss" row, which is meant to test the output side, retains the D=127-dimensional phase head while changing only the loss, which produces a failure specific to that configuration: the remaining D−2 phase dimensions receive zero gradient and drift. Consequently, it cannot be determined whether the gain stems from the SSP encoding of the inputs or from the phase-space output and loss. Could the authors add a configuration with SSP inputs, a two-dimensional displacement head, and a metric displacement loss, to identify the source of the gain?
- In the "Cartesian history" row, graph recovery is unchanged and counterfactual alignment actually improves. Why is it necessary to encode the history as SSPs, and how do the authors interpret this result?
- In Table 3, only C-JEPA uses a maximum of 3 masked objects and a hidden width of 128, whereas the two graph networks use 2 and 256. Since this makes it impossible to tell whether C-JEPA's weaker performance is due to its architecture or to these settings, could the authors explain the reason for this difference and report results with matched settings?
Thank you for your thoughtful review. We have reframed the paper along the lines you suggested.
W1. The paper is now specific in its focus on state representations in dynamics models operating on ground-truth states. The new title is "Spatial Semantic Pointer Representations for State-Based World Models", and the abstract, introduction, and conclusion make clear that states are given. We no longer describe SSP-JEPA as object-centric, reserving that term in the conclusion for full visual pipelines where objects are extracted from video and SSP-JEPA serves as the dynamics predictor. Readout and editing are no longer framed as advantages over coord-GNN, since both operate on given states, and the baseline is now referred to as the C-JEPA predictor on states. We kept "world model" in the sense of a learned transition model on states.
W2. The abstract quoted the baselines at h = 34 (0.58 and 0.68) and SSP-JEPA at h = 20. The CLEVRER numbers labelled h = 20 were also from h = 54. Both are corrected. Translation and insertion are removed, since only deletion was evaluated. We checked all numbers against the run files and fixed the typos.
Q1. We added the requested control: SSP inputs with a 2-D head and metric loss, re-encoded as a valid SSP at every step. It matches the phase head on both datasets, and a coordinate model using only the SSP edge encoding also matches SSP-JEPA. Thus, the accuracy gain comes from the input-side edge encoding rather than the phase output or loss.
Q2. On springs, we find no evidence that encoding history as SSPs improves performance, as differences remain within seed noise. On CLEVRER, the SSP-edge ablation (which uses raw displacement history) also matches SSP-JEPA. We also now note in the text that the Cartesian-history ablation used a learned identity token.
Q3. Mask size 3 was nominally best among {2,3,4} for C-JEPA (without a statistically significant difference), and width 128 was its default. When evaluated at the GNN settings (mask 2, width 256), C-JEPA does not improve on any springs metric. On CLEVRER, the configuration mismatch went the other way (C-JEPA used mask 2 while the GNNs used 3); increasing C-JEPA to mask 3 yields a modest improvement (ρ(20) from 0.81 to 0.76; contact error from 0.75 to 0.72), but it still trails SSP-JEPA. Both sweeps are included in Appendix B.
Q3. A sweep was performed for optimal number of masks and it was found that C-JEPA performed best with 3, with no significant difference, and width 128 was the default. At the graph networks' settings (mask 2, width 256) C-JEPA is not better on any springs axis.