acceptodds
Under review as a conference paper at ICLR 2027

Learning Persistent World Representations for Spatial Reasoning from Partial Observations

Abstract

Spatial reasoning over videos and multiple views requires integrating spatial evidence distributed across partial observations. However, existing approaches mainly aggregate visual evidence without explicitly constructing a representation of the observed world. We argue that spatial reasoning should be mediated by a query-independent persistent world representation, causally constructed from partial observations before downstream reasoning. Following this formulation, we propose the *Internal Spatial World Encoder* (ISWE), which augments a vision-language model with a causal world-construction process that enables world-informed visual enrichment. ISWE preserves observation-specific evidence in local states and causally integrates it into persistent World representations through asymmetric Local-to-World updates. The constructed world is maintained through complementary temporal and persistent memory carriers, and is accessed to enrich visual tokens by combining local and cross-observation information before query-dependent answer generation. We further propose *Scope-Aligned Spatial Internalization*, which aligns spatial supervision with the validity scopes of local, world, and memory representations. ISWE-8B achieves a macro-average score of **69.6** on VSI-Bench. It further demonstrates strong generalization on ViewSpatial-Bench and MMSI-Bench, achieving **58.8** and **38.3**, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Spatial Intelligence, Latent World Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PrJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Spatial reasoning over multiple views and videos is challenging for vision-language models (VLMs) because it requires maintaining a 3D memory that integrates spatial evidence for every frame. This work presents the Internal Spatial World Encoder (ISWE). Essentially, it augments a VLM with a world construction process, which allows the VLM to condition on and answer the query. It uses a Local-to-World update to convert local information into a world representation consisting of causal world states that recurrently update persistent world memory. It also uses a temporal world bank for frames to attend to previous frames via self-attention. The most interesting part of this paper is distilling 3D features from pretrained foundation models, as teachers, into that persistent world memory. Experiments show that it outperforms existing models including both proprietary and open-sourced models in the VSI-Bench. It also achieves very good results in ViewSpatial-Bench and MMSI-Bench, better than all open-sourced models, and on-par with several proprietary models, but worse than strong multimodal proprietary models like Gemini.

Critical strengths
  1. The method is elegant. I especially like the part where you distill the 3D "capabilities" of those pretrained foundation vision models into your method. Maintaining a temporal world bank and a persistent world memory is also an intuitive design. The memory state of the persistent world memory is well-designed, which uses a learned gate to balance new information against old memory.

  2. The paper is well-written with illustrative figures. These figures help in understanding the paper.

Critical weaknesses
  1. Table 3 shows the performance of the base model SenseNova-SI-1.5 is not large? It only achieves in average 3.1 pp gain compared to the untrained base model and even smaller for LoRA-trained model? How significant is this? Could you show some kind of statistical measures like variance to demonstrate that? Also, you should include the base model performance in the two main results tables.

  2. No ablation results to show the benefit of each module. For example, what's the performance of the model with only persistent world memory or only the temporal world bank?

Questions for authors
  1. Could you make the text of the right-side panel in Figure 1 larger? Currently, I have to zoom in to 500% to see what it says.

Some of the questions are included in the weaknesses. Please answer those.

6 comments
AuthorsOpenReview ↗

We Thank the Reviewer PrJJ for the Constructive Feedback and Valuable Suggestions

Re1:

We thank the reviewer for the concerns regarding performance gains, statistical reliability, and baseline completeness. We first clarify that SenseNova-SI-1.5 is a pretrained model with strong spatial reasoning capabilities, rather than an untrained baseline. ISWE aims to further improve cross-observation spatial reasoning by explicitly constructing and accessing persistent world representations.

To facilitate reproducibility and further evaluation, we have released the model weights and detailed results in our anonymous repository.

The released checkpoint preserves the original Local–World–Memory architecture, persistent world construction mechanism, and core training objectives, with adjustments to training frame configurations and hyperparameters. We will explicitly document these changes and distinguish the original submission results from those of the released checkpoint.

1. Performance Gains on VSI-Bench and VSTI-Bench

Under a unified evaluation protocol, the released ISWE-8B improves VSI-Bench Overall from 66.54 to 70.65 (+4.11pp) over the base model and outperforms LoRA by 2.78pp. Since ISWE primarily targets cross-observation spatial integration, we additionally evaluate VSTI-Bench, where ISWE achieves 69.08, outperforming Base (59.93) and LoRA (65.05) by 9.15pp and 4.03pp, respectively.

VSTI-Bench:

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
SenseNova-SI-1.5-8B (Base)46.4028.1961.6689.7473.6759.93
SenseNova-SI-1.5-8B + LoRA61.6230.7358.1689.2485.5165.05
ISWE-8B (Released)62.9231.9174.4891.9084.1869.08

VSI-Bench:

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderOverall
SenseNova-SI-1.5-8B (Base)68.8754.1272.3865.6367.7581.7045.3676.5466.54
SenseNova-SI-1.5-8B + LoRA71.3051.5775.4469.8066.8983.6847.4576.8367.87
ISWE-8B (Released)73.6357.0576.8275.1069.7285.8544.8582.2070.65

2. Cross-Benchmark Performance Preservation

We further evaluate ISWE on four additional out-of-distribution benchmarks: MMSI, MMSI-Video, ViewSpatial, and SQA3D, under a unified protocol. The released ISWE achieves modest improvements over the base model on all four benchmarks, with no overall performance degradation observed. These results suggest that ISWE largely preserves the baseline's cross-benchmark capabilities while improving video-based spatial reasoning. Given the small gains, we do not claim statistically significant generalization improvements.

3. Statistical Reliability

We acknowledge the reviewer's concern regarding statistical reliability. As additional evidence, our unified evaluations show consistent improvements over Base on VSI-Bench (+4.11pp) and VSTI-Bench (+9.15pp), as well as over LoRA on VSTI-Bench (+4.03pp). While these results support the effectiveness of ISWE, they do not replace multi-seed statistical analysis. We will also include the SenseNova-SI-1.5 baseline results across additional benchmarks in the revised experimental tables.

AuthorsOpenReview ↗

We Thank the Reviewer PrJJ for the Constructive Feedback and Valuable Suggestions

Re2:

We thank the reviewer for the constructive feedback and suggestions for further analyzing our architecture. We first clarify that ISWE enhances visual features through two complementary pathways: Local Access retrieves observation-specific spatial evidence from Local Banks, while Memory Access retrieves accumulated cross-observation information from Persistent World Memory. Together, they provide complementary spatial evidence for downstream reasoning. To further assess their contributions, we conduct the following experiments.

1. Additional Ablations on VSI-Bench

Using the latest released ISWE-8B checkpoint (70.65), we independently disable Local Access and Memory Access while keeping all trained parameters and other inference settings fixed.

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65
w/o Memory Access70.9955.9473.9973.3865.4683.3342.8579.2368.15
w/o Local Access71.0355.8874.2773.8765.6183.0343.3679.3968.30
Nearest-Video Memory72.1355.6675.4173.4868.0284.2543.3180.3869.08
Zero-readout71.4853.2174.6671.0567.4279.9642.9382.3367.88

Removing Memory Access and Local Access reduces Overall accuracy by 2.50pp and 2.35pp, respectively. Nearest-Video Memory and Zero-readout yield drops of 1.57pp and 2.77pp. Notably, Zero-readout disables the contributions of both access pathways to visual enhancement while retaining the remaining architecture and trained parameters. These results support the complementary contributions of local spatial evidence and accumulated world information.

2. Additional Validation on VSTI-Bench

We further apply the same interventions to VSTI-Bench, which emphasizes cross-observation spatial relationships and camera motion.

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Full ISWE62.9231.9174.4891.9084.1869.08
w/o Memory Access59.4530.7871.5189.9883.0666.96
w/o Local Access59.3630.6071.2990.0383.1566.89
Nearest-Video Memory58.4831.9271.0586.4085.6866.71
Zero-readout57.6128.6070.1785.5983.8765.17

Removing Memory Access and Local Access reduces Overall accuracy by 2.12pp and 2.19pp, respectively. Nearest-Video Memory and Zero-readout further reduce accuracy to 66.71 and 65.17. These consistent drops across both benchmarks provide additional evidence for the contributions of the two access pathways.

3. Parameter-Matched Question-Guided Attention Comparison

To further evaluate the architecture, we introduce a parameter-matched Question-Guided Attention baseline that replaces ISWE, while retaining the same teacher supervision and training settings. Full ISWE outperforms this baseline by 2.96pp on VSI-Bench and 2.17pp on VSTI-Bench, supporting the effectiveness of its recurrent world construction and complementary visual access mechanisms.

VSI-Bench

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Attention72.3051.4073.4470.5667.4181.7345.8478.8367.69
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65

VSTI-Bench

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Attention59.4732.3872.0188.9281.7766.91
Full ISWE62.9231.9174.4891.9084.1869.08

Re3:

Thank you for this helpful suggestion. We agree and have enlarged the text in the right-hand panel of Fig. 1, adjusting the layout to improve readability at normal viewing size.

AuthorsOpenReview ↗

Follow-up on Revised Results and Controlled Ablation Studies

Dear Reviewer,

Thank you again for your insightful comments and constructive suggestions. We have submitted our revised manuscript, incorporating expanded baseline comparisons, additional controlled ablations, and clearer experimental analyses.

In particular, we have strengthened the evaluation on VSI-Bench and VSTI-Bench, clarified the individual contributions of Local Access and Persistent World Memory, and included a parameter-matched Attention Adapter comparison to better assess the architectural contribution of ISWE.

We sincerely hope these revisions provide more convincing evidence for the effectiveness of our approach. If you have any remaining concerns or would like further clarification regarding the experimental results, we would be very happy to continue the discussion.

We would be grateful if you could reconsider your evaluation in light of the revised results and additional analyses.

Thank you once again for your careful consideration and valuable feedback.

Best regards,
The Authors

Reviewer_MMCvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Internal Spatial World Encoder (ISWE), a framework for spatial reasoning. The key idea is to construct a query-independent persistent world representation from partial observations before downstream reasoning. It also introduces Scope-Aligned Spatial Internalization that aligns spatial supervision with the validity scopes of local, world, and memory representations. The proposed method achieves leading performance over several spatial benchmarks.

Critical strengths
  1. The idea of separating query-independent spatial representation construction from downstream reasoning is an interesting and potentially useful way to improve spatial reasoning.

  2. The proposed method achieves strong performance across several benchmarks, outperforming previous methods.

Critical weaknesses
  1. The writing of the paper is poor and hard to follow.

  2. Although the authors claim that they construct a query-independent persistent latent world representation, the paper provides limited evidence that the latent representation captures spatial consistency across observations. Furthermore, the model is trained with only 12 uniformly sampled frames per video, which may provide insufficient spatial coverage. This raises concerns about whether the proposed method can learn a coherent and persistent representation under such sparse observations.

  3. The paper lacks sufficient experiments to support the effectiveness of and justify the proposed design. In Table 4, removing the Memory contribution from the visual readout results in a modest performance drop, raising concerns about its actual contribution.

  4. The qualitative examples in Fig. 3 seem to need to be checked to see whether the answers and the views match, and it would be better if the targets were highlighted with red bounding boxes.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

We Thank the Reviewer MMCv for the Constructive Feedback and Valuable Suggestions

We thank the reviewer for the valuable suggestions. To further substantiate the effectiveness and reproducibility of ISWE, we have publicly released the model weights and detailed evaluation results in our anonymous repository.

Re 1:

We thank the reviewer for the careful assessment and constructive feedback. To address the clarity concerns, we have clarified ISWE's core idea and processing flow, refined the method descriptions, and revised Figure 2. We will also thoroughly revise the manuscript to improve its clarity and readability.

1. Clarification of the Core Idea and Processing Flow

The central idea of ISWE is to construct a query-independent persistent world representation, Wt=(L1:t,Mt)W_t=(L_{1:t},M_t), by causally integrating partial observations before the downstream question qq is introduced into the frozen language model. This representation is then used to enhance visual features for spatial reasoning.

As illustrated in Figure 2, ISWE consists of three interconnected stages. First, Local-to-World Update extracts observation-specific spatial information and transfers it to the World stream through unidirectional information flow. Next, the Temporal World Bank causally integrates spatial information across observations through attention, interleaved with local updates. Finally, Persistent World Memory consolidates these world states into a fixed-capacity memory through gated recurrent updates. The resulting representation retains both Local Banks, which preserve observation-specific evidence, and Persistent World Memory, which provides accumulated spatial context for subsequent visual enhancement.

2. Revisions to Improve Clarity

We have revised the manuscript to clarify these mechanisms. In Sections 3.1–3.2, we distinguish the roles of Local, World, and Memory representations. The Temporal World Bank serves as an intermediate state for causal world construction, while Local Banks and Persistent World Memory form the visual access interface. We also clarify the distinction between interleaved Local–World updates and recurrent memory updates across observations.

In Section 3.3, we explain the complementary roles of Snapshot Access and Memory Access. Both use queries derived from visual features, while the textual question is introduced only during final answer generation by the frozen language model.

In Section 3.4 and Figure 2, we reorganize teacher supervision according to the information scopes of Local, World, and Memory. We also revise the information-flow labels and figure caption to clarify the relationships among world construction, memory updates, and visual readout.

AuthorsOpenReview ↗

We Thank the Reviewer MMCv for the Constructive Feedback and Valuable Suggestions

Re2:

We thank the reviewer for highlighting the insufficient experimental evidence for cross-observation spatial consistency. To address this concern, we conduct the following additional experiments to evaluate whether ISWE supports consistent spatial reasoning across observations.

1. Spatial Fact Retention under Additional Observations

We select 500 early-frame spatial relation questions from 90 VSTI-Bench scenes. Keeping the questions, target frames, and ground-truth answers fixed, we extend the input from the first 8 to all 32 sampled frames to examine whether additional observations affect previously established spatial judgments.

ModelAcc. (8 frames)Acc. (32 frames)Correct Retention
SenseNova-SI-1.5-8B (Base)85.2084.80397/416 (95.43%)
Full ISWE91.0090.00408/416 (98.08%)
w/o ISWE Path85.2084.80397/416 (95.43%)

ISWE outperforms the baseline by 5.80pp and 5.20pp under 8-frame and 32-frame inputs, respectively. Among the 416 questions correctly answered by all three settings with 8 frames, ISWE retains 98.08% correct answers after adding observations, compared with 95.43% for both baselines (+2.64pp). These results suggest that ISWE better preserves previously correct spatial judgments when integrating additional observations.

2. Spatial Consistency across Non-overlapping Observations

We further evaluate 250 paired spatial relation questions (500 instances) from 89 scenes. Each pair shares the same question, answer choices, and ground truth but uses different target frames. We divide 32 sampled frames into two non-overlapping 16-frame sets, each containing its corresponding target frame, with no duplicated RGB images across sets.

ModelObs. A Acc.Obs. B Acc.OverallAnswer ConsistencyBoth CorrectConsistent but Wrong
SenseNova-SI-1.5-8B (Base)85.6090.4088.0082.4080.402.00
Full ISWE88.8093.6091.2088.8085.603.20
w/o Memory Access86.4091.2088.8086.0084.002.00

Compared with the baseline, ISWE improves answer consistency from 82.40% to 88.80% (+6.40pp) and both-correct accuracy from 80.40% to 85.60% (+5.20pp), increasing correctly answered pairs from 201/250 to 214/250. Removing Memory Access reduces consistency to 86.00% (−2.80pp), supporting the contribution of Persistent World Memory to cross-observation consistency.

3. Sparse-Observation Training

The original submission uses 12 uniformly sampled frames per video during training to control computational cost. Similar limited-frame sampling strategies are adopted by SpatioLM, GeoMan, CALM, and Bridging Information Asymmetry. Moreover, Appendix B.2 shows that increasing inference frames from 4 to 32 improves ISWE's accuracy from 57.18% to 71.75% on a fixed 500-question VSI-Bench subset. This indicates that ISWE can benefit from additional observations beyond its training frame count.

Together, these experiments provide further evidence that ISWE supports consistent spatial reasoning across observations and can preserve previously correct spatial judgments while incorporating additional visual evidence.

AuthorsOpenReview ↗

We Thank the Reviewer MMCv for the Constructive Feedback and Valuable Suggestions

Re3:

We thank the reviewer for raising concerns about the effectiveness of Persistent World Memory. We first clarify that ISWE enhances visual features through two complementary pathways: Local Access retrieves observation-specific spatial evidence from Local Banks, while Memory Access retrieves accumulated cross-observation information from Persistent World Memory. Together, they provide complementary spatial evidence for downstream reasoning.

The Zero Memory ablation in the original Table 4 primarily intervenes in the Memory branch while retaining Local Access and the learned parameters of the visual access modules. Thus, its performance drop reflects the additional contribution of Memory beyond local spatial enhancement, rather than the full contribution of the ISWE architecture. To better isolate both pathways, we conduct additional controlled ablations using the latest released ISWE-8B checkpoint, keeping model parameters and other inference settings fixed.

1. Additional Ablations on VSI-Bench

We separately disable Local Access and Memory Access to evaluate their respective contributions.

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65
w/o Memory Access70.9955.9473.9973.3865.4683.3342.8579.2368.15
w/o Local Access71.0355.8874.2773.8765.6183.0343.3679.3968.30
Nearest-Video Memory72.1355.6675.4173.4868.0284.2543.3180.3869.08
Zero-readout71.4853.2174.6671.0567.4279.9642.9382.3367.88

Removing Memory Access reduces Overall from 70.65 to 68.15 (−2.50pp), while removing Local Access reduces it to 68.30 (−2.35pp). Nearest-Video Memory and Zero-readout further yield 69.08 (−1.57pp) and 67.88 (−2.77pp), respectively. Notably, Zero-readout disables the visual enhancement contributions of both Local and Memory Access while retaining the remaining model architecture and trained parameters. These results support the complementary contributions of local evidence and persistent world information.

2. Additional Validation on VSTI-Bench

We extend the same four interventions to VSTI-Bench, which emphasizes cross-observation spatial relationships and camera motion.

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Full ISWE62.9231.9174.4891.9084.1869.08
w/o Memory Access59.4530.7871.5189.9883.0666.96
w/o Local Access59.3630.6071.2990.0383.1566.89
Nearest-Video Memory58.4831.9271.0586.4085.6866.71
Zero-readout57.6128.6070.1785.5983.8765.17

Removing Memory Access reduces Overall from 69.08 to 66.96 (−2.12pp), while removing Local Access reduces it to 66.89 (−2.19pp). Nearest-Video Memory and Zero-readout further reduce Overall to 66.71 (−2.37pp) and 65.17 (−3.91pp), respectively. These results further support the contributions of both access pathways on cross-observation spatial reasoning tasks. We will incorporate these ablations and clarify their experimental settings and interpretations in the revised manuscript.

Re4:

We thank the reviewer for the suggestion. We have added red bounding boxes, consistent object instance IDs across frames, and timestamps to Fig.3. These annotations highlight the relevant objects and allow readers to directly examine how the predicted answers relate to the corresponding video views.

Reviewer_yY9mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

he paper argues that spatial reasoning over video and multi-view input should go through a query-independent, persistent latent "world" representation, built from partial observations before the question is seen. It proposes the Internal Spatial World Encoder (ISWE), a trainable side module of about 424M parameters on a frozen SenseNova-SI-1.5-InternVL3-8B. ISWE has four parts:

  • Local states. Each frame has its own Local state, refined over J=3J=3 interleaved steps. Local states write one way into a World stream.
  • World stream. A causally masked self-attention stream, the "Temporal World Bank" B1:tB_{1:t}.
  • Persistent World Memory. A gated recurrent memory MtM_t with 8 Layout and 8 Trajectory tokens.
  • World-to-Visual Access. Each frame's visual tokens attend to their own Local Bank (64 tokens per frame) and to the final Memory. The enriched tokens are then injected into six layers of the frozen LM.

Training adds "Scope-Aligned Spatial Internalization" to the answer loss. This is about 12 distillation and regression terms from Grounding DINO, VGGT and LingBot-Map, each assigned to the Local, World or Memory representation. The teachers are not used at inference.

The model is trained on 111K VLM-3R samples for one epoch with one seed. It reports 69.6 on VSI-Bench, 58.8 on ViewSpatial-Bench and 38.3 on MMSI-Bench. The paper also reports:

  • A controlled comparison on VSI-Bench using the same backbone and data: base 66.5, LoRA 67.9, reproduced SpatioLM 68.0, ISWE without auxiliary losses 68.0, full ISWE 69.6.
  • On MMSI, the same base model at 36.9 against ISWE at 38.3.
  • An inference-time "Zero Memory" diagnostic.
  • Appendix studies on interleaving depth, the readout interface, injection density, channel attribution and inference cost.
Critical strengths
  1. A well-motivated design with a controlled gain (main Table 3; App. Tables 8 and 10). The design separates per-frame evidence (Local), a causal cross-frame stream (World) and a fixed-size recurrent Memory, and assigns each teacher to the representation whose scope matches it. This is clearly motivated, and to my knowledge this specific combination is new. The evaluation is also unusually controlled for this area: most work compares methods across different backbones, data and frame budgets, but here the backbone, the training subset, the optimizer and the 48-frame VSI-Bench protocol are fixed. ISWE is compared with the unadapted base model, LoRA (r=48) and a reproduction of SpatioLM. Under these conditions it gains +3.1 macro-average on VSI-Bench over the base model and +1.6 to +1.7 over SpatioLM and LoRA, and it is ahead of LoRA on all eight tasks. This is the main reason for my positive rating.

  2. Teacher-free inference with detailed documentation. Three teachers are distilled during training and none is needed at inference. App. Table 11 reports about 3x lower latency than VGGT-based spatial MLLMs (1.29 vs. about 3.9 s/sample), with no runtime 3D model. The appendix documents feature taps, every loss weight, the memory update rule, the training configuration and several ablations, which helps reproducibility.

Critical weaknesses
  1. The headline numbers are not reported against the backbone, and as presented they overstate the contribution.
  • MMSI-Bench. The public model card of SenseNova-SI-1.5-InternVL3-8B reports 38.3, the same as ISWE's 38.3. Against the authors' own re-run of the base model (36.9), the gain is +1.4 points on 1,000 questions, which is within one standard error.
  • ViewSpatial-Bench. The same model card reports 59.0 for the base model; ISWE scores 58.8. The abstract's "strong generalization on ViewSpatial-Bench" therefore describes the backbone. The authors re-ran the base model with lmms-eval for MMSI but not for ViewSpatial.
  • VSI-Bench. Table 1 claims the highest macro-average but leaves out the base model (66.5 in the authors' run), Cambrian-S-7B (67.5), and the published SpatioLM result on a SenseNova-SI 8B backbone (71.6, ICML 2026), which is higher than ISWE's 69.6.
  • The VLM2 per-task scores in App. Table 9 do not match the public VLM2 paper (arXiv 2511.20644, Table 1). With the public numbers, VLM2 is ahead on absolute distance, relative direction and route planning, which reverses the conclusions in App. C.3.

After these corrections, the solid result is the controlled +3.1 on VSI-Bench. These are problems with how the results are reported, not with the method, but the corrections are required.

  1. The mechanism behind the gains is not isolated.
  • Architecture alone gives no measurable gain. "ISWE (answer loss only)" scores 68.0, the same as LoRA (67.9) and the SpatioLM reproduction (68.0). The extra gain comes from the teacher losses, which are never applied to LoRA or to a plain adapter of matched size, and the scope assignment is never tested on its own. Distillation alone is known to help (3DRS, Spatial Forcing, SpatioLM), and SpatioLM already uses a very similar frozen-backbone side module. The parts that are new, the recurrent Memory and the scope assignment, are therefore the ones that need evidence.
  • The Memory's measured contribution is small. Zeroing it at inference (Table 4) changes the reported tasks by at most 1.4 points. The overall score is not reported, and Nearest-Video Memory and Zero-readout are shown only as examples.
  • Every result comes from a single seed, while the differences behind the design choices are 0.5 to 1.6 points.

Overall: I lean toward acceptance because the controlled VSI-Bench gain is real and the design is well motivated and well documented. This rating assumes the revision (i) adds the backbone and the missing baselines to Tables 1-2 and revises the abstract accordingly, (ii) corrects the VLM2 numbers in App. Table 9, and (iii) reports seed variance for the main controlled comparison. If the rebuttal cannot address these, I would lower my score.

Questions for authors

Questions that could change my assessment:

Q1. Please add to Tables 1-2: SenseNova-SI-1.5 under your protocol on all three benchmarks (including ViewSpatial with lmms-eval), Cambrian-S-7B, and the published SpatioLM result (71.6), with the protocol differences explained. If the ViewSpatial and MMSI gains over the backbone do not hold, please revise the abstract and Sec. 4.2.

Q2. Which source produced the VLM2 per-task row in App. Table 9? It does not match the public paper. Please correct the table and App. C.3.

Q3. Please report at least three seeds for full ISWE, ISWE (answer loss only), LoRA and the SpatioLM reproduction, with paired significance tests.

Q4. Please apply the same teacher losses to LoRA and to a parameter-matched plain adapter, and train a variant with the same teachers but no scope assignment. If these reach about 69.6, the gain comes from distillation rather than from world construction.

Q5. Please report Zero Memory, Nearest-Video Memory and Zero-readout on the full benchmark (all eight tasks and the overall score).

4 comments
AuthorsOpenReview ↗

We Thank the Reviewer for the Constructive Feedback and Valuable Suggestions

Re 1: The headline numbers are not reported against the backbone, and as presented they overstate the contribution.

We thank the reviewer for highlighting the incomplete baseline comparisons and the need for more precise generalization claims. To further validate ISWE's effectiveness and reproducibility, we have released the model weights and detailed evaluation results in our anonymous repository.

The released checkpoint preserves the original Local–Memory architecture, persistent world construction mechanism, and core training objectives, with refinements to the training frame configuration and hyperparameters. We will adopt the released checkpoint results in the revised manuscript.

Under a unified evaluation protocol, ISWE improves over SenseNova-SI-1.5-8B from 66.54 to 70.65 (+4.11pp) on VSI-Bench and from 59.93 to 69.08 (+9.15pp) on VSTI-Bench. Compared with SpatioLM re-evaluated using its official checkpoint, ISWE achieves comparable performance on VSI-Bench (70.65 vs. 70.43) and a larger numerical advantage on VSTI-Bench (69.08 vs. 55.97, +13.11pp ).

ISWE also achieves modest positive gains across four OOD benchmarks (MMSI, MMSI-Video, ViewSpatial, and SQA3D), without an observed decline in overall performance. We therefore limit our generalization claims to cross-benchmark performance preservation and modest improvements, without overstating statistical significance.

ModelVSI-BenchVSTI-BenchMMSI (OOD)MMSI-Video (OOD)ViewSpatial (OOD)SQA3D (OOD)
SenseNova-SI-1.5-8B (Base)66.5459.9336.9026.7659.0239.53
ISWE-8B (Released)70.6569.0837.6027.6759.2340.38
Improvement (pp)+4.11+9.15+0.70+0.91+0.21+0.85

1. VSTI-Bench: Additional Baseline Comparisons

We extend the VSTI-Bench comparison to SpatioLM, VLM², and Cambrian-S. ISWE achieves the highest overall accuracy among the evaluated methods.

ModelCam-Obj Abs. Dist.Cam. Displace.Cam. Mov. Dir.Obj-Obj Rel. Pos.Cam-Obj Rel. Dist.Avg.
SenseNova-SI-1.5-8B46.4028.1961.6689.7473.6759.93
SpatioLM-Understanding-8B45.7931.1355.4271.5175.9955.97
VLM²-7B43.1044.1076.8087.7074.9065.30
Cambrian-S-7B33.5025.6145.5665.1782.8350.54
ISWE-8B (Released)62.9231.9174.4891.9084.1869.08

2. MMSI-Bench and ViewSpatial

On MMSI-Bench, the baseline and ISWE are evaluated under identical hardware, inference precision, tile settings, decoding, and scoring protocols. The released ISWE improves accuracy from 36.90% to 37.60% (+0.70pp). On ViewSpatial, both models are evaluated using lmms-eval with matched hardware, precision, decoding, and scoring protocols. ISWE achieves 59.23%, compared with 59.02% for the baseline.

These results provide additional evidence that ISWE preserves baseline capabilities on these OOD benchmarks while achieving modest numerical gains.

3. VSI-Bench: Controlled Baseline Comparisons

We further evaluate Cambrian-S-7B and SpatioLM using the same hardware, inference precision, decoding, frame count, and scoring protocol as ISWE.

MethodObject CountingAbsolute DistanceObject SizeRoom SizeRelative DistanceRelative DirectionRoute PlanningAppearance OrderAvg.
Cambrian-S-7B61.9643.8668.8961.0166.4869.9535.5775.8960.45
SpatioLM-Understanding-8B71.8658.1878.2574.5570.0085.8447.4277.3570.43
ISWE-8B (Released)73.6357.0576.8275.1069.7285.8544.8582.2070.65

4. VLM²: Corrected Reporting and Re-evaluation

We have corrected the VLM² results cited from the original paper and re-evaluated VLM² under matched hardware, inference precision, decoding, frame count, and scoring settings. We use the re-evaluated results for our main analysis while retaining the published scores for reference.

VSI-Bench TaskVLM² (Published)VLM² (Re-evaluated, 48 frames)ISWE-8B
Object Counting72.5072.0273.63
Absolute Distance59.6061.0957.05
Object Size70.8069.4576.82
Room Size69.9069.5175.10
Relative Distance69.0066.2069.72
Relative Direction87.8082.9485.85
Route Planning52.6050.5244.85
Appearance Order68.3075.8982.20
Overall (Macro Avg.)68.8068.4570.65
AuthorsOpenReview ↗

We Thank the Reviewer for the Constructive Feedback and Valuable Suggestions

Re 2: The mechanism behind the gains is not isolated.

We thank the reviewer for the constructive feedback.

(1) Architecture alone gives no measurable gain.

To directly evaluate the architectural contribution, we introduce a parameter-matched Question-Guided Attention baseline that replaces ISWE, while retaining the same teacher supervision and training settings. Full ISWE outperforms this baseline by 2.96pp on VSI-Bench and 2.17pp on VSTI-Bench, supporting the effectiveness of its recurrent world construction and complementary visual access mechanisms.

VSI-Bench:

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Attention Adapter72.3051.4073.4470.5667.4181.7345.8478.8367.69
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65

VSTI-Bench:

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Attention Adapter59.4732.3872.0188.9281.7766.91
Full ISWE62.9231.9174.4891.9084.1869.08

We further compare ISWE with Base and LoRA on VSTI-Bench.

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
SenseNova-SI-1.5-8B (Base)46.4028.1961.6689.7473.6759.93
SenseNova-SI-1.5-8B + LoRA61.6230.7358.1689.2485.5165.05
ISWE-8B (Released)62.9231.9174.4891.9084.1869.08

Ablation of Scope-Aligned Supervision.

To assess the contribution of different supervision sources, we separately remove Grounding DINO, VGGT Depth/XYZ, and LingBot-Map losses while keeping the remaining architecture and training settings unchanged.

VSI-Bench:

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65
w/o Grounding-DINO Loss73.0351.9975.5771.0468.4583.7748.9779.1368.99
w/o VGGT Depth/XYZ Loss74.0252.9175.6672.8168.1785.1148.4577.0269.27
w/o LingBot-Map Loss73.0852.5275.8671.8467.8986.1846.3976.8668.83

VSTI-Bench:

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Full ISWE62.9231.9174.4891.9084.1869.08
w/o Grounding-DINO Loss59.4029.1664.2090.6886.0365.89
w/o VGGT Depth/XYZ Loss59.6029.9964.9691.1687.1066.56
w/o LingBot-Map Loss59.6130.0964.3191.6386.9566.52

(2) The Memory's measured contribution is small.

ISWE combines Local Access for observation-specific evidence and Memory Access for cross-observation information. The original Zero Memory ablation retains Local Access, measuring Memory's additional contribution. We further isolate both pathways through inference-time ablations on the released ISWE-8B checkpoint with all other settings fixed.

VSI-Bench:

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppear. OrderAvg.
Full ISWE73.6357.0576.8275.1069.7285.8544.8582.2070.65
w/o Memory Access70.9955.9473.9973.3865.4683.3342.8579.2368.15
w/o Local Access71.0355.8874.2773.8765.6183.0343.3679.3968.30
Nearest-Video Memory72.1355.6675.4173.4868.0284.2543.3180.3869.08
Zero-readout71.4853.2174.6671.0567.4279.9642.9382.3367.88

VSTI-Bench:

We further apply the same interventions to VSTI-Bench, which emphasizes cross-observation spatial relationships and camera motion.

ModelAbs. Dist.Displace.Mov. Dir.Obj-Obj Rel. Pos.Rel. Dist.Avg.
Full ISWE62.9231.9174.4891.9084.1869.08
w/o Memory Access59.4530.7871.5189.9883.0666.96
w/o Local Access59.3630.6071.2990.0383.1566.89
Nearest-Video Memory58.4831.9271.0586.4085.6866.71
Zero-readout57.6128.6070.1785.5983.8765.17

(3) Single seed.

We acknowledge the single-seed limitation due to the high cost of full training. Under a unified evaluation protocol, ISWE outperforms the baseline by 4.11pp on VSI-Bench and 9.15pp on VSTI-Bench, and LoRA by 4.03pp on VSTI-Bench, providing complementary evidence without establishing statistical significance.

AuthorsOpenReview ↗

Revised Manuscript — Expanded Baselines and Additional Controlled Evidence

Dear Reviewer,

We sincerely appreciate your rigorous assessment and detailed constructive feedback. We have now submitted our revised manuscript, with particular attention to the concerns you raised regarding baseline completeness, the interpretation of performance gains, and the contributions of our architectural design.

We have expanded the comparisons under a unified evaluation protocol, revised our cross-benchmark claims to avoid overstating generalization, and added a parameter-matched Attention Adapter comparison and further ablation studies to better distinguish architectural contributions from auxiliary supervision.

Your feedback has been especially valuable in helping us improve the rigor and transparency of our experimental presentation. We sincerely hope these revisions address your principal concerns. If any questions remain unresolved, we would greatly appreciate the opportunity to discuss them further.

We would also be grateful if you could reassess the paper in light of the revised manuscript and new experimental evidence.

Thank you again for your time and careful consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Additional Downstream Navigation Results and Architectural Evidence

Dear Reviewer yY9m,

We sincerely appreciate your rigorous and constructive review, particularly your recognition of our controlled comparisons and the motivation behind the Local–World–Memory architecture.

Following our detailed rebuttal and revised manuscript, we have conducted an additional closed-loop ObjectNav experiment on HM3D v0.2 to examine whether the improvements in spatial reasoning can translate into downstream embodied decision-making.

Using the same 200 episodes from all 36 validation scenes, the same VLMNav navigation policy, and the same two-GPU configuration, we obtain:

MetricSpatioLMISWE
Success Rate3.5% (7/200)14.0% (28/200)
SPL (%)0.825.32
Avg. Time / Episode20.82 s8.16 s

Thus, ISWE achieves a +10.5pp navigation success-rate improvement (4×) and approximately 2.55× evaluation throughput relative to SpatioLM.

These results complement our previous evidence:

  • Controlled spatial reasoning: ISWE improves over Base by +9.15pp on VSTI-Bench and +4.11pp on VSI-Bench.
  • Architectural contribution: Against a parameter-matched Attention Adapter trained with the same teacher supervision, ISWE improves by 2.96pp on VSI and 2.17pp on VSTI.
  • Memory contribution: Removing Memory Access reduces performance by 2.50pp on VSI and 2.12pp on VSTI.

We have also corrected the baseline comparisons and reporting issues highlighted in your review, and revised our generalization claims to avoid overstatement.

We hope that the new navigation experiment, together with the controlled architectural comparisons and corrected reporting, provides stronger evidence for both the effectiveness and practical relevance of the proposed representation.

We sincerely appreciate your careful assessment and would be grateful if these additional findings could be considered in your final evaluation.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Experimental Update: Substantial Gains in Closed-Loop Navigation

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate your time and efforts throughout the review process. As the author–reviewer discussion period approaches its conclusion, we would like to share one final experiment that provides new evidence of ISWE's downstream practical effectiveness.

1. New Closed-Loop Navigation Evaluation

We conducted a closed-loop ObjectNav evaluation on HM3D v0.2, sampling 200 episodes across all 36 validation scenes. To ensure a controlled comparison, ISWE and SpatioLM were evaluated on identical episodes using the same VLMNav navigation policy and two-GPU configuration.

MetricSpatioLMISWE
Successful Episodes7/20028/200
Success Rate3.5%14.0%
SPL (%)0.825.32
Avg. Time / Episode20.82 s8.16 s

ISWE achieves a +10.5pp improvement in navigation success rate (4×), substantially higher SPL, and approximately 2.55× evaluation throughput.

Importantly, this experiment extends our evaluation beyond spatial question answering to closed-loop embodied navigation, where spatial understanding must support sequential decisions. While limited to a controlled 200-episode subset, it provides additional evidence of ISWE's downstream utility.

2. Summary of Previous Rebuttal Evidence

Our earlier revisions and additional experiments have also demonstrated:

  • Spatial reasoning: +9.15pp on VSTI-Bench and +4.11pp on VSI-Bench over the backbone.
  • Architectural effectiveness: +2.96pp on VSI and +2.17pp on VSTI over a parameter-matched, teacher-matched Attention Adapter.
  • Persistent memory: Removing Memory Access reduces performance by 2.50pp on VSI and 2.12pp on VSTI.
  • Cross-observation consistency: Answer consistency improves by 6.40pp over the backbone under non-overlapping observations.

Together, these experiments provide complementary evidence across spatial reasoning benchmarks, controlled architectural analyses, cross-observation consistency, and downstream navigation.

3. Request for Final Consideration

We have made every effort to address the reviewers' concerns through substantial experiments, detailed responses, and a revised manuscript.

We respectfully hope that the newly demonstrated closed-loop navigation improvements, together with the full body of revised experimental evidence, can be carefully considered during the final assessment.

We would also be grateful if the remaining reviewers could be encouraged to consider our revisions, as we have not yet received their follow-up feedback.

Thank you very much for your continued time, understanding, and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.