acceptodds
Under review as a conference paper at ICLR 2027

Probing Spatial State Representations in Visual Foundation Models

Abstract

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore concerns more than recognizing individual images: it calls for a persistent spatial state that can be updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We investigate this question through four levels of spatial capability: spatial structure, object persistence, predictive spatial update, and actionability, using frozen feature probes across ten representations under streaming egocentric observations in static indoor environments. The resulting capability profiles vary across tasks, network layers, and history lengths: strong visible scene readout does not ensure accurate object persistence or future feature retrieval. Matched controls further distinguish readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure but changes its interface to a fixed spatial readout. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: in the continuous output setting, predicted actions respond to goal changes, yet using the correct goal does not reliably improve progress. Together, these findings motivate evaluating spatial representations not only by what can be decoded, but by whether that information can be maintained beyond observation, predictively updated, and used for goal-directed action.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Intelligence, Visual foundation models, Spatial state representations, Probing, Object persistence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uCb4OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors characterize spatial state representations through four levels: spatial structure, object persistence, predictive spatial update, and actionability. Across ten frozen representations, readouts vary across tasks and extraction settings. Complementary analyses of motion, history, prediction, and action distinguish decodability from input use and functional utility, supporting multidimensional evaluation beyond a single spatial task score.

Critical strengths
  1. The L1–L4 architecture proposed in this paper provides a comprehensive framework for evaluating the spatial perception capabilities of visual foundation models.
  2. The paper draws several insightful conclusions.
  3. The paper designs rigorous controlled experiments for history, motion, predictive readout, and goal use.
Critical weaknesses
  1. The model employs an approach based on independently encoded causal prefixes, which does not exhibit an incremental, recurrent state update. It is recommended that this methodological boundary be more prominently highlighted in the paper, along with a brief discussion of how this framework might be extended in the future to architectures with explicit recurrent states.

  2. Current experiments are primarily limited to offline, self-centered trajectories in static indoor environments (ScanNet++), excluding object dynamics, contact physics, or more complex open physical worlds. The authors should more explicitly highlight the generalization limits of the existing framework in open-world or dynamic environments in the discussion or conclusion section.

  3. For generative models (such as Wan and CogVideoX) and world models, their performance is highly sensitive to specific generation coordinates or layer selections. How do the authors ensure that the selected representative layers and scheduler steps (e.g., layer 20 and step 749 in Wan) possess sufficient robustness, rather than being merely coincidental results under a specific hyperparameter setting?

  4. Since different models retrieve future features within their respective target spaces in prediction tasks (such as one-step prediction and path integration), these scores are not universal physical error metrics. When conducting cross-model comparisons, how do the authors ensure that differences in scale across different feature spaces do not bias conclusions?

Questions for authors

Please see the Critical Weaknesses section.

3 comments

Thank you for the positive assessment and the constructive suggestions.

W1. Independently encoded prefixes versus recurrent state updates

We agree. The submission states this boundary in Sec. 2.2 and Appendix C, and the revision makes it prominent in the Introduction and Discussion. Each prefix is encoded independently, so the question is what the representation of a causal prefix affords, not whether the model maintains an online state. New history interventions fit this boundary. On the same prefix-12 probes as Table 3, removing the target-visible frames or all visual tokens raises V-JEPA2's and Wan's localization error by 6.6–13.6° (intervals above zero), whereas shuffling the historical frames changes the error by at most 0.95° and reversing them by at most 2.15° (V-JEPA2, the only interval excluding zero); the probes rely on historical content and only weakly on its order.

Extension to explicit recurrent states. For models that carry a state across frames, such as streaming reconstruction models with a persistent state (CUT3R, Spann3R) or recurrent and state-space video models, the same four heads can read the state after each update instead of a re-encoded prefix. This enables three tests our current setting cannot make: (i) state versus prefix, i.e., whether the updated state supports L1–L4 as well as re-encoding the history does; (ii) update consistency, i.e., whether kk incremental updates reach the state obtained from the kk-step prefix; and (iii) order sensitivity, applying the shuffle and reverse interventions to the update sequence. Our path-integration probe and the external recurrent updater of the motion-composition study are first steps in this direction, since both impose a recurrent update on frozen features. We added this discussion to the conclusion.

W2. Generalization beyond static indoor scenes

Agreed. The revision adds a limitations paragraph to the conclusion: our data contain observer motion through static indoor scenes, and object dynamics, contact physics, outdoor or large-scale environments, and closed-loop embodied interaction are outside the evaluated scope, so our results should not be read as statements about them. The four levels themselves are not specific to static scenes, but extending them changes what each level demands. For moving objects, L2 requires predicting where an unseen object has moved, and L3 must predict object motion as well as ego-motion; both require data with dynamic 3D object tracks. We list these as the main directions for extending the benchmark.

W3. Robustness of layer and generation-coordinate choices (e.g., Wan block 20 at step 749)

We separate conclusions that hold across extraction settings from those that do not, using the full sweeps of the submission (Figs. 4a, 6 and 7; Appendix A.2).

  • No post-hoc maxima in the main table. The designated layers are fixed per model (for Wan and CogVideoX, block 20 at scheduler step 749, following VidFM3D), and the sweeps are analyzed separately. The designated layer is often not the best one -- Wan's sweeps peak at block 15 for geometry, view consistency, persistence and path integration, and V-JEPA2's at blocks 17, 11 or 5 rather than its designated block 23 (Fig. 4a).
  • Regularities that hold across settings. In three of the four multi-layer backbones (VideoMAE, V-JEPA2, Wan), persistence peaks at the same block as metric geometry; a longer prefix worsens persistence and path integration for every model with both endpoints; and Wan favors the earlier generation coordinate on depth, view consistency and persistence in all 12 layer–prefix pairs (Appendix A.2).
  • Setting-dependent results are flagged. At block 20, instance identity favors the later coordinate for Wan but the earlier one for CogVideoX, and in VideoMAE, V-JEPA2 and Wan one-step prediction peaks at a different block from persistence (17 vs 23, 5 vs 17, 10 vs 15), so no single layer represents all readouts.

We therefore present layer and coordinate results as profiles rather than single-setting rankings, and the revision states which conclusions depend on the setting.

W4. Cross-model comparison of prediction scores in different feature spaces

We agree that raw retrieval scores are not comparable across feature spaces, as Appendix C of the submission notes, and the new controls show how much this matters. For the eight models of the main prediction run, the evaluation of the Table 2 predictors also includes a copy-last-observation reference and no-action and shuffled-action inputs on the same queries and gallery (full table in our response to Reviewer kpWg). The ordering by raw one-step R@1 closely follows the copy ordering (ρ=0.88\rho = 0.88), so raw R@1 largely reflects how retrievable a target is in each space. The revision therefore draws cross-model L3 conclusions only from within-space references, namely gains over copying and over wrong or absent motion, and the matched-wrong-motion test of Sec. 4.3, never from raw R@1. On this basis, learned one-step prediction exceeds copying on average only for Fast3R and CogVideoX, at low absolute levels.This keeps scale differences between feature spaces out of our cross-model conclusions.

We thank you again for the suggestions, which made the paper clearer about its scope and more careful in its comparisons.

Thank you for the comprehensive response, and I maintain my rating.

Reviewer_1s66OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors evaluate 10 visual foundation models' ability to spatially perceive their environment beyond single observations. They evaluate these models on four criteria: spatial structure (L1), object persistence (L2), predictive spatial update (L3), and actionability (L4). They find mixed success across models, with no single model performing unilaterally well across all four criteria L1-L4.

Critical strengths

I think the L1-L4 criteria are quite sensible measures of spatial representation beyond single observation, and are plausibly comprehensive. This is an important prerequisite for having a strong benchmark.

Critical weaknesses
  • The criteria do not have a ceiling notion that's clearly articulated, and it is unclear how these criteria relate to prior work. For the same reason, the observations used (ScanNet++) are not clear why they ought to be adequate or novel.

  • The models are not frontier, even the open-source models are not frontier (e.g. Qwen3-VL would be good to evaluate, as well as measuring behavioral versions of L1-L4 in closed-source models).

Questions for authors
  • What is the gold-standard ceiling in each L1-L4 criteria? What would qualify as "good"?

  • The models are primarily open-source models which don't necessarily represent the frontier. For example, I see Qwen-2.5-VL is used but what about the Qwen-3-VL series? Furthermore, are there ways to adapt the measures (e.g. just performance on your four criteria) to report numbers on closed-source frontier models? Do they do much better than open-source models on the same performance criteria?

  • Is the data only restricted to ScanNet++ trajectories? Why is that a good choice for this benchmark? I didn't really see a justification for this choice. It would've been good to see some examples of these images in the main and/or supplement, which I didn't see.

  • There's no related work in the main text, and what is in the supplement section B is unclear how it builds on existing benchmarks and how it differentiates or advances them. There are clear analogues of L1-L4 individually in prior work, and these need to be properly cited and differentiated.

  • The takeaways are unclear, as it seems the results are that there is mixed performance across the models. What types of models should the community invest in next based on your benchmark? And what, if anything about your benchmark allows you to conclude that over prior vision benchmarks?

  • Minor: the notion of a "prefix" could be better defined, and why is it significant/relevant for spatial reasoning? (It seems to me a bit a priori strange for a spatial representation benchmark)

7 comments

Thank you for the thoughtful questions. We ran new experiments where a question could be answered empirically and revised the paper where it concerns presentation.

Q1. What is the ceiling at each level, and what qualifies as "good"?

Each metric has a natural bound (zero loss or error, AUC@30 =1= 1, balanced accuracy =1= 1, R@1 =1= 1), but a frozen-probe score is limited by both the representation and the probe, so a fixed numeric ceiling would largely reflect the probe. We therefore define "good" relative to anchors measured with the same probe, data and split, and the revision reports every level against them:

LevelAnchors a representation must exceedReference pointsWhere the best models stand
L1 structurechance (view consistency: 0.5 balanced accuracy)a 3D-reconstruction model (Fast3R)view consistency is near its bound (0.983), but camera-pose AUC@30 reaches only 0.385
L2 persistencethe same probe with history removed: current frame only, target-visible frames removed, visual tokens zeroed0° errorbest error 55.5° (VideoMAE); of the four models tested, only V-JEPA2 and Wan show a reliable benefit from history
L3 predictioncopy of the last observation; no-action and shuffled-action inputs; matched wrong motionR@1 =1= 1learned one-step prediction exceeds copying only for Fast3R and CogVideoX, at R@1 below 0.03
L4 actionaction-only scorer (67.8%); shuffled goal; goal-only and current-only inputsan endpoint reference that observes realized outcomes (68.3–85.5%)candidate selection reaches 75.9–79.9%, but in the continuous setting the correct goal adds nothing over the current-only input for any of the four models

A representation is "good" at a level if it exceeds every anchor of that level, and a good spatial-state representation would do so at all four levels. The anchors also show how far current models are from this: camera pose is far from its bound, history dependence at L2 is reliable for only two of the four models tested, most learned predictors do not beat copying at L3, and goal information brings no benefit in continuous L4. In this sense the benchmark is far from saturated, and each level exposes a distinct gap.

Q2. Frontier models: Qwen3-VL and closed-source models

Our ten representations already include models released in 2025–2026 (V-JEPA 2, Wan2.1, Fast3R, Aether, MiniMax-H3, and the 30B-parameter LingBot-video MoE). Following your suggestion, we added Qwen3-VL-8B-Instruct as a new experiment, using its visual-merger output with the same 12×1212\times 12 token grid per frame as Qwen2.5-VL. To isolate the effect of the newer model, we ran it in a paired comparison with Qwen2.5-VL-7B, with the same six probes, hyperparameters, split and seed for both.

ReadoutQwen3-VL relative to Qwen2.5-VL
Depth loss ↓−14%
Camera-pose AUC@30 ↑+20%
Instance-identity loss ↓−14%
View-consistency accuracy ↑+0.9 points
Object-persistence error ↓+2.5° (worse; per-prefix intervals overlap)
Path-integration R@1 ↑−0.013

Qwen3-VL improves every scene-content readout, but not object persistence or path integration. Its camera-pose readout (AUC@30 =0.032= 0.032) stays an order of magnitude below Fast3R's in Table 2 (0.385), its persistence error still grows with a longer prefix (+11.2° from prefix 8 to 24), and, as for Qwen2.5-VL, its one-step predictor stays below the copy reference. This is a concrete instance of the paper's main finding: a stronger multimodal encoder improves scene-content readout, but not the pose-linked persistence that a spatial state requires.

Closed-source models. Frozen-feature probing needs internal activations, which closed-source APIs do not expose, so L1–L4 cannot be measured at the representation level for them. A behavioral version (e.g., asking where a previously seen object is now) measures the system's language-mediated answer, which mixes the representation with reasoning and answer formatting; such behavioral evaluations of proprietary models already exist in spatial question-answering benchmarks such as VSI-Bench. Our study is complementary: it asks what spatial information a representation exposes and uses under matched controls, which a behavioral score cannot separate. The revision states this distinction in the Related Work and lists a behavioral counterpart of L1–L4 as future work.

Q3. Why ScanNet++, and examples

The four levels need, in the same real-world data: (i) continuous egocentric trajectories with accurate camera poses, to form causal prefixes, ego-motion actions, waypoints and goal frames; (ii) metric geometry, for depth, camera-pose and view-overlap labels; (iii) 3D instance annotations, to define instance identity and to decide when a previously seen object has left the view and where it is relative to the current camera; and (iv) static scenes, matching our scope of observer motion. ScanNet++ provides laser-scanned geometry registered to handheld captures and dense 3D instance annotations of real indoor scenes, so all labels come from one consistent geometry; our streaming benchmark uses 612 training and 69 validation scenes (276 validation windows per prefix). Real captures matter because the probed models were trained on real images, and synthetic scenes would add a rendering domain gap. The pipeline needs only posed RGB-D video with 3D instance annotations, so it carries over to other datasets; relying on one dataset is a limitation we now state explicitly. Following your suggestion, the revision adds an appendix figure with example windows showing the prefix frames, the hidden target and its bearing, and the future and goal frames.

Reviewer_kpWgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines spatial representations in ten frozen visual foundation models using egocentric sequences from ScanNet++. It probes spatial structure, object persistence, motion-conditioned prediction, and goal-conditioned action. Through layer/prefix analyses and input controls, the authors find that strong spatial readout does not necessarily translate into effective use of history or motion, nor into useful goal-directed actions.

Critical strengths
  • The evaluation goes beyond geometric probing (e.g., VidFM3D) to object persistence, prediction, and action. I think this scope is useful: recovering visible geometry alone gives an incomplete picture of what a spatial representation can support.
  • The input controls are the strongest part of the paper. Copying features, removing history, substituting motion, and shuffling goals help figure out what drives performance. In particular, separating changes in predicted actions from actual progress toward the goal makes the action results much more informative.
Critical weaknesses
  • The cross-model prediction comparisons are hard to interpret for me. Each model retrieves future targets in its own feature space, so a higher R@1 may reflect an easier retrieval problem rather than better spatial prediction. The paper acknowledges this limitation, but Section 4.1 still draws comparisons from these rankings. Moreover, the copying and matched-motion controls cover only a subset of models, so the leading DINOv2 and Qwen2.5-VL results remain insufficiently checked.
  • Key parts of the main benchmark are underspecified. Probe architectures, feature pooling, training budgets, and task-specific sample counts for Table 2 are not documented well enough to judge whether the comparisons are fair. This limits confidence in reproducibility and generality.
Questions for authors
  1. Which reported results use scenes that were untouched by earlier probe training and experimental development? How does the overlap in Section 3.1 relate to the task-specific scene-disjoint splits?

  2. What probe architectures, feature pooling methods, and training budgets are used for Table 2? How is probe capacity controlled across models with different feature dimensions and token counts?

  3. For DINOv2 and Qwen2.5-VL, how much do one-step prediction and path-integration scores improve over copying and incorrect-motion controls?

I think the paper proposes an interesting benchmark, and I will reconsider the score if my questions are fully addressed.

5 comments

Thank you for the careful review and the precise questions. We ran new analyses for each of them.

Q3 / W1. Do the DINOv2 and Qwen2.5-VL prediction scores survive copying and incorrect-motion controls?

New analysis. The evaluation of the Table 2 predictors also scored, at fixed weights and on the same queries and gallery: (i) no action input, (ii) a shuffled action (another query's motion), and (iii) a copy reference that retrieves with the pooled feature of the last observed prefix frame. Sec. 4.3 reported these only for V-JEPA2 and Wan at prefix 12. Below are all eight models of the main prediction run.

ModelOne-step (Table 2)No actionShuffled actionCopy last obs.
DINOv20.45920.29530.27810.5172
Qwen2.5-VL0.43750.38770.38950.5453
V-JEPA20.12320.04800.06610.2038
Wan0.11500.06790.09150.1350
VideoMAE0.08880.04440.05340.1386
CogVideoX0.02720.01900.02080.0172
Fast3R0.02630.02080.02630.0072
Aether0.01090.01180.01180.0181

Answer for DINOv2 and Qwen2.5-VL.

Versus incorrect or absent motion. Both predictors use the supplied motion. DINOv2 gains +0.181+0.181 over the shuffled action and +0.164+0.164 over no action; Qwen2.5-VL gains +0.048+0.048 and +0.050+0.050. Both gains over the shuffled action are positive at every prefix. For path integration the stored evaluation has no wrong-motion input; there the copy comparison below is the decisive test, since both models stay far below copying.

Versus copying. Neither exceeds it, at any prefix. DINOv2 reaches 0.459 against 0.517 for copying, and Qwen2.5-VL 0.438 against 0.545. For path integration, the stored evaluation has no copy reference, so we ran a new experiment in which path-integration probes and a copy reference are scored against the same 828 waypoint candidates: both models then fall about 0.16 R@1 below copying (−0.159-0.159 and −0.165-0.165).

DINOv2's and Qwen2.5-VL's predictors therefore use motion, but do not move their output far enough to beat the identity map in global retrieval. Their high raw R@1 mainly reflects that adjacent frames lie close together in these two feature spaces, so the correct target is easy to retrieve from the last frame. This is the distinction drawn in Appendix B.1 of the submission: strong future-feature retrieval is not credited as predictive update unless it passes matched controls. V-JEPA2 shows the same pattern, consistent with Sec. 4.3, where its correct-motion gain was the largest (18.8 points) while copying rivaled learned prediction.

W1. Cross-model comparison when each model retrieves in its own feature space

We agree that raw R@1 should not be compared across feature spaces as a measure of predictive update; Sec. 3.2 of the submission notes that these scores are not a common scale of prediction error, and the new controls now quantify the difference. Across the eight models above, the ordering by one-step R@1 in Table 2 closely follows the ordering by the copy reference (Spearman ρ=0.88\rho = 0.88; path integration also gives ρ=0.88\rho = 0.88 against the same copy ordering): raw R@1 largely measures how easily a target can be retrieved in a given feature space. The revision therefore:

  • reports one-step prediction with its copy, no-action and shuffled-action references for every model, and bases cross-model statements about predictive update on these within-space differences and on the matched-wrong-motion test of Sec. 4.3, not on raw R@1;
  • states in Sec. 4.1 that the lead of DINOv2 and Qwen2.5-VL is a lead in future-feature retrieval, their predictors use motion but do not exceed copying (answer to Q3). Averaged over prefixes, learned one-step prediction exceeds copying only for Fast3R (+0.019+0.019) and CogVideoX (+0.010+0.010), both at R@1 below 0.03. Wan exceeds copying at prefix 12, the comparison reported in Sec. 4.3 (15.22% vs 11.96%), but not on average over prefixes (−0.020-0.020);
  • extends the caveat of Appendix C, that retrieval scores in different feature spaces are not a shared error scale, with this measured reference.

One further caveat, which the revision states: for encoders that process the whole prefix jointly, the copy uses the last frame's in-context feature, whereas targets are encoded in isolation. Overall, these analyses reinforce the paper's central point that decodability, input use and functional utility have to be measured separately.

Q2 / W2. Probe architectures, pooling, training budgets, and capacity control

The revision adds a full specification (new appendix with a table of probe heads). In brief:

TaskInput and poolingHeadLossEpochs
Metric geometrytoken grids of all prefix frames, projected to 1,024 channelsVidFM3D's probe: 4 attention blocks, DPT depth head and camera headdepth and camera-pose losses as in VidFM3D100
Instance identityas abovesame trunk, trained separately; DPT head with 64-d pixel embeddingsprototype cross-entropy100
View consistencyspatially pooled feature per frameLayerNorm–MLP C→512→256C \to 512 \to 256; pair MLP on [zi, zj, zi⊙zj, ∣zi−zj∣][z_i,\ z_j,\ z_i \odot z_j,\ \lvert z_i - z_j\rvert]binary cross-entropy on overlap labels50
Object persistencetokens of all prefix frames after 2×22\times 2 pooling, projected to 512object query + 4-layer Transformer, 8 headscross-entropy over 16×816\times 8 azimuth–elevation bins + smooth-L1 on log distance50
One-step predictionpooled feature per prefix frame + 9-D action token2-layer Transformer, width 512, 8 heads; MLP to CCMSE +(1−cos⁡)+ (1 - \cos)50
Path integrationpooled prefix feature; one 9-D action per waypointtwo stacked GRU cells (width 512); decoder to CCMSE +(1−cos⁡)+ (1 - \cos)50

Q2 / W2.

Training budget. AdamW (β=0.9/0.95\beta = 0.9/0.95, weight decay 0.05), linear warm-up then cosine decay to 1e−61\mathrm{e}{-6}, bf16, seed 42. Geometry and instance identity use learning rate 1e−41\mathrm{e}{-4} for 100 epochs (10 warm-up epochs); the other probes use 3e−43\mathrm{e}{-4} for 50 epochs (5 warm-up epochs). The batch size is one window for geometry, instance identity and persistence and four for the other probes, with a few backbone-specific exceptions. Every model, layer and prefix is trained on windows from the same 612 training scenes.

Capacity control. Every head first projects the model's CC-dimensional features to a fixed width (1,024 or 512). Everything after this projection, including depth, width, attention heads, optimizer and epochs, is identical across models, layers and prefixes; only the input projections (and, for the two prediction heads, the output layer back to CC) depend on CC. Token count changes how many tokens a head attends over, not its depth or width: geometry and instance probes attend over each model's native grid (e.g., 16×1616\times 16 for V-JEPA2, 30×5230\times 52 for Wan) and upsample to the 288×512288\times 512 label resolution, persistence uses 2×22\times 2-pooled tokens, and the other probes pool over space.

Sample counts for Table 2 (per prefix, validation windows from 69 scenes). 276 windows for geometry, instance identity and view consistency; 67, 165, 169 and 169 windows with an eligible hidden target for persistence at prefixes 8, 12, 16 and 24; 276 queries for one-step prediction; and 276 windows, each scored at three waypoints, for path integration.

Q1. Which results use scenes untouched by probe training and development, and how does this relate to the Sec. 3.1 sentence?

In short, every reported number is computed on scenes that no probe of that experiment was trained on, but none is computed on a split that was untouched by experimental development.

  • The dataset has one scene-level split with disjoint training, validation and test scene lists. The submitted Sec. 3.1 sentence explains why we did not report on the historical test partition: it overlapped manifests from earlier development, so it could not serve as an independent test. No result uses it.
  • Table 2 and the layer and prefix analyses train on windows from 612 training scenes and evaluate on windows from 69 validation scenes. The motion and history controls (Sec. 4.3 and the analyses above) use the same validation scenes, as does the native-prediction readout. Candidate-selection actions use 289 training and 77 validation scenes, and continuous actions 444 and 108, each scene-disjoint.
  • The validation scenes are not a blind test: they also served as our development set while the tasks were designed (the motion-composition study, for example, evaluates new frame triples from scenes used in its pilot). To limit selection effects, Table 2 uses designated layers fixed per model and averages all evaluated prefixes rather than reporting best-layer or best-prefix values, and the probe architectures and optimizer settings are shared across models.
  • Fast3R's training data include ScanNet++, and some of our validation scenes are among its training scenes, so its leading camera-pose and candidate-selection results may partly reflect familiarity with these scenes.

The revised Sec. 3.1 states these facts explicitly and describes all results as held-out validation evidence rather than blind-test evidence. We hope these analyses address your questions, and we are glad to provide more.

Comments on the paper

1 comment

Summary of new experiments and analyses

We thank all reviewers for their careful reading and constructive questions. Since submission we have run new experiments and analyses aimed at the shared concerns: how prediction scores compare across feature spaces, which inputs the probes actually use, how the four levels relate, and which models are covered. These additions complement the submitted results.

  • Controls for every one-step predictor of Table 2. The evaluation of these predictors also scored, at fixed weights, a copy-last-observation reference and no-action and shuffled-action inputs at every prefix; Sec. 4.3 reported them only for V-JEPA2 and Wan at prefix 12. We now report all eight models of the main prediction run, and add a copy reference for path integration in a new experiment.
  • Complete history interventions for object persistence on the same prefix-12 probes as Table 3: target-visible frames removed, visual tokens zeroed, and shuffled or reversed history, each with paired scene-bootstrap intervals.
  • A new model, Qwen3-VL-8B-Instruct, compared with Qwen2.5-VL-7B in a paired run (see our response to Reviewer 1s66).
  • A cross-level analysis of Table 2 that relates object persistence and prediction to the L1 readouts.
  • Documentation and framing: a full probe specification (architectures, pooling, losses, optimizer, epochs, sample counts), Related Work in the main text, window and prefix definitions with example windows, and an explicit limitations paragraph.

Two findings from these analyses sharpen the paper's message.

  1. Object persistence follows camera-pose readout. In Table 2 the three most accurate localizers (VideoMAE, Fast3R, V-JEPA2) are exactly the three best camera-pose readers, and across the ten representations persistence is associated with camera pose (Spearman ρ=0.61\rho = 0.61) more than with depth (0.45), view consistency (0.41) or instance identity (0.02). Qwen3-VL follows this pattern: it improves every scene-content readout over Qwen2.5-VL, but not persistence.
  2. Predictive update beyond copying is rare. How easily the next frame can be retrieved from the last observed one differs strongly between feature spaces. The ordering of one-step R@1 in Table 2 follows that of the copy reference (ρ=0.88\rho = 0.88), and learned one-step prediction exceeds copying, on average over prefixes, only for Fast3R and CogVideoX, both at R@1 below 0.03. We therefore report L3 against these references.

Both results support the paper's central point that decodability, input use and functional utility must be measured separately. We respond to each reviewer individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.