- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Spatial reasoning over multiple views and videos is challenging for vision-language models (VLMs) because it requires maintaining a 3D memory that integrates spatial evidence for every frame. This work presents the Internal Spatial World Encoder (ISWE). Essentially, it augments a VLM with a world construction process, which allows the VLM to condition on and answer the query. It uses a Local-to-World update to convert local information into a world representation consisting of causal world states that recurrently update persistent world memory. It also uses a temporal world bank for frames to attend to previous frames via self-attention. The most interesting part of this paper is distilling 3D features from pretrained foundation models, as teachers, into that persistent world memory. Experiments show that it outperforms existing models including both proprietary and open-sourced models in the VSI-Bench. It also achieves very good results in ViewSpatial-Bench and MMSI-Bench, better than all open-sourced models, and on-par with several proprietary models, but worse than strong multimodal proprietary models like Gemini.
Critical strengths
-
The method is elegant. I especially like the part where you distill the 3D "capabilities" of those pretrained foundation vision models into your method. Maintaining a temporal world bank and a persistent world memory is also an intuitive design. The memory state of the persistent world memory is well-designed, which uses a learned gate to balance new information against old memory.
-
The paper is well-written with illustrative figures. These figures help in understanding the paper.
Critical weaknesses
-
Table 3 shows the performance of the base model SenseNova-SI-1.5 is not large? It only achieves in average 3.1 pp gain compared to the untrained base model and even smaller for LoRA-trained model? How significant is this? Could you show some kind of statistical measures like variance to demonstrate that? Also, you should include the base model performance in the two main results tables.
-
No ablation results to show the benefit of each module. For example, what's the performance of the model with only persistent world memory or only the temporal world bank?
Questions for authors
- Could you make the text of the right-side panel in Figure 1 larger? Currently, I have to zoom in to 500% to see what it says.
Some of the questions are included in the weaknesses. Please answer those.
6 comments
We Thank the Reviewer PrJJ for the Constructive Feedback and Valuable Suggestions
Re2:
We thank the reviewer for the constructive feedback and suggestions for further analyzing our architecture. We first clarify that ISWE enhances visual features through two complementary pathways: Local Access retrieves observation-specific spatial evidence from Local Banks, while Memory Access retrieves accumulated cross-observation information from Persistent World Memory. Together, they provide complementary spatial evidence for downstream reasoning. To further assess their contributions, we conduct the following experiments.
1. Additional Ablations on VSI-Bench
Using the latest released ISWE-8B checkpoint (70.65), we independently disable Local Access and Memory Access while keeping all trained parameters and other inference settings fixed.
| Model | Obj. Count | Abs. Dist. | Obj. Size | Room Size | Rel. Dist. | Rel. Dir. | Route Plan | Appear. Order | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Full ISWE | 73.63 | 57.05 | 76.82 | 75.10 | 69.72 | 85.85 | 44.85 | 82.20 | 70.65 |
| w/o Memory Access | 70.99 | 55.94 | 73.99 | 73.38 | 65.46 | 83.33 | 42.85 | 79.23 | 68.15 |
| w/o Local Access | 71.03 | 55.88 | 74.27 | 73.87 | 65.61 | 83.03 | 43.36 | 79.39 | 68.30 |
| Nearest-Video Memory | 72.13 | 55.66 | 75.41 | 73.48 | 68.02 | 84.25 | 43.31 | 80.38 | 69.08 |
| Zero-readout | 71.48 | 53.21 | 74.66 | 71.05 | 67.42 | 79.96 | 42.93 | 82.33 | 67.88 |
Removing Memory Access and Local Access reduces Overall accuracy by 2.50pp and 2.35pp, respectively. Nearest-Video Memory and Zero-readout yield drops of 1.57pp and 2.77pp. Notably, Zero-readout disables the contributions of both access pathways to visual enhancement while retaining the remaining architecture and trained parameters. These results support the complementary contributions of local spatial evidence and accumulated world information.
2. Additional Validation on VSTI-Bench
We further apply the same interventions to VSTI-Bench, which emphasizes cross-observation spatial relationships and camera motion.
| Model | Abs. Dist. | Displace. | Mov. Dir. | Obj-Obj Rel. Pos. | Rel. Dist. | Avg. |
|---|---|---|---|---|---|---|
| Full ISWE | 62.92 | 31.91 | 74.48 | 91.90 | 84.18 | 69.08 |
| w/o Memory Access | 59.45 | 30.78 | 71.51 | 89.98 | 83.06 | 66.96 |
| w/o Local Access | 59.36 | 30.60 | 71.29 | 90.03 | 83.15 | 66.89 |
| Nearest-Video Memory | 58.48 | 31.92 | 71.05 | 86.40 | 85.68 | 66.71 |
| Zero-readout | 57.61 | 28.60 | 70.17 | 85.59 | 83.87 | 65.17 |
Removing Memory Access and Local Access reduces Overall accuracy by 2.12pp and 2.19pp, respectively. Nearest-Video Memory and Zero-readout further reduce accuracy to 66.71 and 65.17. These consistent drops across both benchmarks provide additional evidence for the contributions of the two access pathways.
3. Parameter-Matched Question-Guided Attention Comparison
To further evaluate the architecture, we introduce a parameter-matched Question-Guided Attention baseline that replaces ISWE, while retaining the same teacher supervision and training settings. Full ISWE outperforms this baseline by 2.96pp on VSI-Bench and 2.17pp on VSTI-Bench, supporting the effectiveness of its recurrent world construction and complementary visual access mechanisms.
VSI-Bench
| Model | Obj. Count | Abs. Dist. | Obj. Size | Room Size | Rel. Dist. | Rel. Dir. | Route Plan | Appear. Order | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Attention | 72.30 | 51.40 | 73.44 | 70.56 | 67.41 | 81.73 | 45.84 | 78.83 | 67.69 |
| Full ISWE | 73.63 | 57.05 | 76.82 | 75.10 | 69.72 | 85.85 | 44.85 | 82.20 | 70.65 |
VSTI-Bench
| Model | Abs. Dist. | Displace. | Mov. Dir. | Obj-Obj Rel. Pos. | Rel. Dist. | Avg. |
|---|---|---|---|---|---|---|
| Attention | 59.47 | 32.38 | 72.01 | 88.92 | 81.77 | 66.91 |
| Full ISWE | 62.92 | 31.91 | 74.48 | 91.90 | 84.18 | 69.08 |
Re3:
Thank you for this helpful suggestion. We agree and have enlarged the text in the right-hand panel of Fig. 1, adjusting the layout to improve readability at normal viewing size.
Follow-up on Revised Results and Controlled Ablation Studies
Dear Reviewer,
Thank you again for your insightful comments and constructive suggestions. We have submitted our revised manuscript, incorporating expanded baseline comparisons, additional controlled ablations, and clearer experimental analyses.
In particular, we have strengthened the evaluation on VSI-Bench and VSTI-Bench, clarified the individual contributions of Local Access and Persistent World Memory, and included a parameter-matched Attention Adapter comparison to better assess the architectural contribution of ISWE.
We sincerely hope these revisions provide more convincing evidence for the effectiveness of our approach. If you have any remaining concerns or would like further clarification regarding the experimental results, we would be very happy to continue the discussion.
We would be grateful if you could reconsider your evaluation in light of the revised results and additional analyses.
Thank you once again for your careful consideration and valuable feedback.
Best regards,
The Authors
We Thank the Reviewer PrJJ for the Constructive Feedback and Valuable Suggestions
Re1:
We thank the reviewer for the concerns regarding performance gains, statistical reliability, and baseline completeness. We first clarify that SenseNova-SI-1.5 is a pretrained model with strong spatial reasoning capabilities, rather than an untrained baseline. ISWE aims to further improve cross-observation spatial reasoning by explicitly constructing and accessing persistent world representations.
To facilitate reproducibility and further evaluation, we have released the model weights and detailed results in our anonymous repository.
The released checkpoint preserves the original Local–World–Memory architecture, persistent world construction mechanism, and core training objectives, with adjustments to training frame configurations and hyperparameters. We will explicitly document these changes and distinguish the original submission results from those of the released checkpoint.
1. Performance Gains on VSI-Bench and VSTI-Bench
Under a unified evaluation protocol, the released ISWE-8B improves VSI-Bench Overall from 66.54 to 70.65 (+4.11pp) over the base model and outperforms LoRA by 2.78pp. Since ISWE primarily targets cross-observation spatial integration, we additionally evaluate VSTI-Bench, where ISWE achieves 69.08, outperforming Base (59.93) and LoRA (65.05) by 9.15pp and 4.03pp, respectively.
VSTI-Bench:
VSI-Bench:
2. Cross-Benchmark Performance Preservation
We further evaluate ISWE on four additional out-of-distribution benchmarks: MMSI, MMSI-Video, ViewSpatial, and SQA3D, under a unified protocol. The released ISWE achieves modest improvements over the base model on all four benchmarks, with no overall performance degradation observed. These results suggest that ISWE largely preserves the baseline's cross-benchmark capabilities while improving video-based spatial reasoning. Given the small gains, we do not claim statistically significant generalization improvements.
3. Statistical Reliability
We acknowledge the reviewer's concern regarding statistical reliability. As additional evidence, our unified evaluations show consistent improvements over Base on VSI-Bench (+4.11pp) and VSTI-Bench (+9.15pp), as well as over LoRA on VSTI-Bench (+4.03pp). While these results support the effectiveness of ISWE, they do not replace multi-seed statistical analysis. We will also include the SenseNova-SI-1.5 baseline results across additional benchmarks in the revised experimental tables.