- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies persistent spatial understanding in multimodal large language models when revisiting indoor environments, focusing on viewpoint changes, ambiguity among similar rooms, and object state changes across sessions. It introduces ReSpaceBench, containing 3,244 questions from 590 scenes across nine tasks covering room association, persistent state maintenance, and downstream reasoning. The authors evaluate 32 models and three video-understanding agent frameworks, and conduct human evaluation, room-association error analysis, and frame-sampling experiments. They also construct RS-30K and fine-tune Qwen3-VL-8B, improving overall accuracy from 35.5% to 60.4%.
The topic has practical relevance. Establishing a controlled revisitation protocol around physical room identity provides a clear and worthwhile focus. The dataset, model evaluations, and training resources constitute a relatively comprehensive benchmark effort.
Critical strengths
1.The research question is clear, and the evaluation protocol is well targeted. Using physical room identity as the basis for association across visits, while distinguishing entering a room from seeing it in the field of view, reduces ambiguity in the definition of a visit. Including rooms of the same category as competing identities also reduces shortcuts based solely on semantic categories.
2.Task coverage is relatively comprehensive. The nine tasks cover room recognition, visit counting, object deduplication, state changes, spatial topology, and temporal ordering, enabling investigation of the relationship between identity association and subsequent question answering.
3.Data generation has a traceable factual basis. Answers are derived from scene metadata, actual visit sequences, and object state records, with visibility checks and manual review. This makes the labels easier to verify than question–answer pairs written entirely through unrestricted manual annotation.
Critical weaknesses
A.Section 4 and Appendix A.1 describe default sampling at 1 fps, followed by further uniform downsampling when a model’s context limit is exceeded. The appendix lists frame limits of 120, 150, 320, or 480 for some models. Table 2 directly compares their accuracy even though they may receive different amounts of visual evidence. Without comparisons under a common frame budget, it is difficult to distinguish differences in model capability from differences in input information.
B.Section 3 and Appendix Table 7 define T7 as question answering about object changes and location history across sessions. Table 2 reports aggregate T7 accuracy, but the detailed error analysis in Section 4.2 mainly concerns T1 and agent trajectories. T7 results conditional on correct room and object identification are not reported. Since T7 requires identifying the room and object before comparing their states, an incorrect final answer cannot be attributed specifically to state-updating difficulty when the initial identity association is already wrong.
Questions for authors
A.How many frames does each model actually receive, and what proportion of key visit events do these frames cover? Under a common frame budget, do the main rankings in Table 2 and the conclusion that spatially specialized models perform relatively poorly still hold?
B.What proportions of T7 errors arise from incorrect room or object identity association versus incorrect state comparison? How much would T7 accuracy improve if the correct room IDs were provided?
2 comments
Dear Reviewer UGea,
Thank you again for your detailed and constructive comments. Your insightful suggestions have helped us strengthen the paper and improve the depth and rigor of our empirical analysis. We have updated the paper to incorporate the additional experiments, along with clarifications regarding the evaluation protocol and the sources of spatial memory errors.
We hope these new results address your concerns. We sincerely appreciate your valuable feedback and would be happy to answer any further questions or provide additional clarifications.
Response to Reviewer UGea
We thank the reviewer for their constructive feedback and recognition of our well-defined evaluation protocol, comprehensive task coverage, and reliable data generation process.
We thank the reviewer for raising this concern. In our original evaluation, videos are sampled at 1 fps, with further downsampling only when required by model-specific context limits. Each room visit includes a 6–9-second look-around within the room, providing approximately 6–9 observations under the full 1-fps setting.
To ensure a fair comparison, we additionally evaluated 11 representative models on ReSpaceBench-mini using the same 128 uniformly sampled frames in Tab. R6. All models use 640 × 480 resolution, except SenseNova-SI-1.5-8B, which uses 420 × 420 to accommodate all 128 frames within its visual token budget. The rank of the models remain generally consistent, with GPT-5.6 Sol and Seed 2.0 Pro retain the top two positions, while the three spatially specialized models remain near the bottom. Although some intermediate rankings change, the results suggest that the relatively poor performance of spatially specialized models cannot be attributed solely to differences frame counts. We will incorporate this controlled comparison into the revised manuscript and clarify the frame-budget and resolution settings in the evaluation protocol.
Table R6: Evaluation under 128-frames on ReSpaceBench-mini
* denotes the models using a capped visual input
Thank you for your suggestion for this experiment. We agree that T7 errors may arise from incorrect room or object identity association or subsequent state changes. Inspired by your suggestion, in Tab. R7, we conducted an identity-oracle experiment on all 559 T7 questions with strongest evaluated model GPT-5.6 Sol to disentangle these factors, progressively providing ground-truth cues:
Table R7: T7 Oracle Identity Analysis
Providing correct room IDs improves accuracy by 4.9, while additional object-level annotations yield a total gain of 7.0. This confirms that identity association contributes to T7 difficulty. These improvements suggest that both room-level and object-level identity association can affect T7 performance, and that explicitly providing identity correspondence helps models better reason about object states across sessions. We will incorporate these findings into the revised paper.
We sincerely thank the reviewer again for their constructive feedback and valuable suggestions. We hope our additional experiments and clarifications have addressed your concerns, and we would be happy to provide further details if needed.