- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors characterize spatial state representations through four levels: spatial structure, object persistence, predictive spatial update, and actionability. Across ten frozen representations, readouts vary across tasks and extraction settings. Complementary analyses of motion, history, prediction, and action distinguish decodability from input use and functional utility, supporting multidimensional evaluation beyond a single spatial task score.
Critical strengths
- The L1–L4 architecture proposed in this paper provides a comprehensive framework for evaluating the spatial perception capabilities of visual foundation models.
- The paper draws several insightful conclusions.
- The paper designs rigorous controlled experiments for history, motion, predictive readout, and goal use.
Critical weaknesses
-
The model employs an approach based on independently encoded causal prefixes, which does not exhibit an incremental, recurrent state update. It is recommended that this methodological boundary be more prominently highlighted in the paper, along with a brief discussion of how this framework might be extended in the future to architectures with explicit recurrent states.
-
Current experiments are primarily limited to offline, self-centered trajectories in static indoor environments (ScanNet++), excluding object dynamics, contact physics, or more complex open physical worlds. The authors should more explicitly highlight the generalization limits of the existing framework in open-world or dynamic environments in the discussion or conclusion section.
-
For generative models (such as Wan and CogVideoX) and world models, their performance is highly sensitive to specific generation coordinates or layer selections. How do the authors ensure that the selected representative layers and scheduler steps (e.g., layer 20 and step 749 in Wan) possess sufficient robustness, rather than being merely coincidental results under a specific hyperparameter setting?
-
Since different models retrieve future features within their respective target spaces in prediction tasks (such as one-step prediction and path integration), these scores are not universal physical error metrics. When conducting cross-model comparisons, how do the authors ensure that differences in scale across different feature spaces do not bias conclusions?
Questions for authors
Please see the Critical Weaknesses section.
3 comments
W3. Robustness of layer and generation-coordinate choices (e.g., Wan block 20 at step 749)
We separate conclusions that hold across extraction settings from those that do not, using the full sweeps of the submission (Figs. 4a, 6 and 7; Appendix A.2).
- No post-hoc maxima in the main table. The designated layers are fixed per model (for Wan and CogVideoX, block 20 at scheduler step 749, following VidFM3D), and the sweeps are analyzed separately. The designated layer is often not the best one -- Wan's sweeps peak at block 15 for geometry, view consistency, persistence and path integration, and V-JEPA2's at blocks 17, 11 or 5 rather than its designated block 23 (Fig. 4a).
- Regularities that hold across settings. In three of the four multi-layer backbones (VideoMAE, V-JEPA2, Wan), persistence peaks at the same block as metric geometry; a longer prefix worsens persistence and path integration for every model with both endpoints; and Wan favors the earlier generation coordinate on depth, view consistency and persistence in all 12 layer–prefix pairs (Appendix A.2).
- Setting-dependent results are flagged. At block 20, instance identity favors the later coordinate for Wan but the earlier one for CogVideoX, and in VideoMAE, V-JEPA2 and Wan one-step prediction peaks at a different block from persistence (17 vs 23, 5 vs 17, 10 vs 15), so no single layer represents all readouts.
We therefore present layer and coordinate results as profiles rather than single-setting rankings, and the revision states which conclusions depend on the setting.
W4. Cross-model comparison of prediction scores in different feature spaces
We agree that raw retrieval scores are not comparable across feature spaces, as Appendix C of the submission notes, and the new controls show how much this matters. For the eight models of the main prediction run, the evaluation of the Table 2 predictors also includes a copy-last-observation reference and no-action and shuffled-action inputs on the same queries and gallery (full table in our response to Reviewer kpWg). The ordering by raw one-step R@1 closely follows the copy ordering (), so raw R@1 largely reflects how retrievable a target is in each space. The revision therefore draws cross-model L3 conclusions only from within-space references, namely gains over copying and over wrong or absent motion, and the matched-wrong-motion test of Sec. 4.3, never from raw R@1. On this basis, learned one-step prediction exceeds copying on average only for Fast3R and CogVideoX, at low absolute levels.This keeps scale differences between feature spaces out of our cross-model conclusions.
We thank you again for the suggestions, which made the paper clearer about its scope and more careful in its comparisons.
Thank you for the comprehensive response, and I maintain my rating.
Thank you for the positive assessment and the constructive suggestions.
W1. Independently encoded prefixes versus recurrent state updates
We agree. The submission states this boundary in Sec. 2.2 and Appendix C, and the revision makes it prominent in the Introduction and Discussion. Each prefix is encoded independently, so the question is what the representation of a causal prefix affords, not whether the model maintains an online state. New history interventions fit this boundary. On the same prefix-12 probes as Table 3, removing the target-visible frames or all visual tokens raises V-JEPA2's and Wan's localization error by 6.6–13.6° (intervals above zero), whereas shuffling the historical frames changes the error by at most 0.95° and reversing them by at most 2.15° (V-JEPA2, the only interval excluding zero); the probes rely on historical content and only weakly on its order.
Extension to explicit recurrent states. For models that carry a state across frames, such as streaming reconstruction models with a persistent state (CUT3R, Spann3R) or recurrent and state-space video models, the same four heads can read the state after each update instead of a re-encoded prefix. This enables three tests our current setting cannot make: (i) state versus prefix, i.e., whether the updated state supports L1–L4 as well as re-encoding the history does; (ii) update consistency, i.e., whether k incremental updates reach the state obtained from the k-step prefix; and (iii) order sensitivity, applying the shuffle and reverse interventions to the update sequence. Our path-integration probe and the external recurrent updater of the motion-composition study are first steps in this direction, since both impose a recurrent update on frozen features. We added this discussion to the conclusion.
W2. Generalization beyond static indoor scenes
Agreed. The revision adds a limitations paragraph to the conclusion: our data contain observer motion through static indoor scenes, and object dynamics, contact physics, outdoor or large-scale environments, and closed-loop embodied interaction are outside the evaluated scope, so our results should not be read as statements about them. The four levels themselves are not specific to static scenes, but extending them changes what each level demands. For moving objects, L2 requires predicting where an unseen object has moved, and L3 must predict object motion as well as ego-motion; both require data with dynamic 3D object tracks. We list these as the main directions for extending the benchmark.