acceptodds
Under review as a conference paper at ICLR 2027

Lost in the Hallway: Benchmarking Persistent Spatial Memory Across Revisits

Abstract

Persistent spatial understanding is essential for embodied agents to operate reliably in environments they repeatedly encounter. It requires recognizing revisited places despite viewpoint changes, distinguishing them from similar-looking rooms, and updating memories as environments change. Yet existing evaluations often leave physical place identity across separated visits implicit. We introduce ReSpaceBench, a benchmark comprising 3,244 questions across 590 synthetic and real-home scenes, organized into 9 tasks spanning room association, persistent state tracking, and downstream reasoning. Controlled walkthroughs define visits by physical room entry and deliberately include same-type room distractors and cross-session object changes. Our evaluation of 32 models and 3 agentic video-understanding frameworks reveals substantial limitations: the strongest evaluated zero-shot model achieves 50.5% overall accuracy and 52.2% on a 450-question subset, compared with 90.1% for human annotators. The evaluated agentic frameworks offer no overall improvement over direct inference with the same base model. Analysis of agent execution traces reveals evidence omissions and incorrect cross-visit room associations as major failure modes, suggesting that iterative evidence seeking alone does not ensure consistent spatial memory. Further analyses expose a gap between recognizing familiar rooms and identifying the correct visited room, while annotation-guided sampling highlights the importance of preserving intervening visits and transitions. We also provide RS-30K, a separate training dataset from synthetic sources with no template, scene, route, or video overlap with the benchmark. Fine-tuning Qwen3-VL-8B raises overall accuracy from 35.5% to 60.4%, with the largest gains in association and improvements extending to real-life scenes. ReSpaceBench and RS-30K provide resources for diagnosing and improving persistent spatial memory across revisits.

Primary area:
datasets and benchmarks
Keywords:
Spatial Understanding, Long Video Understanding, Revisit Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UGeaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies persistent spatial understanding in multimodal large language models when revisiting indoor environments, focusing on viewpoint changes, ambiguity among similar rooms, and object state changes across sessions. It introduces ReSpaceBench, containing 3,244 questions from 590 scenes across nine tasks covering room association, persistent state maintenance, and downstream reasoning. The authors evaluate 32 models and three video-understanding agent frameworks, and conduct human evaluation, room-association error analysis, and frame-sampling experiments. They also construct RS-30K and fine-tune Qwen3-VL-8B, improving overall accuracy from 35.5% to 60.4%.

The topic has practical relevance. Establishing a controlled revisitation protocol around physical room identity provides a clear and worthwhile focus. The dataset, model evaluations, and training resources constitute a relatively comprehensive benchmark effort.

Critical strengths

1.The research question is clear, and the evaluation protocol is well targeted. Using physical room identity as the basis for association across visits, while distinguishing entering a room from seeing it in the field of view, reduces ambiguity in the definition of a visit. Including rooms of the same category as competing identities also reduces shortcuts based solely on semantic categories.

2.Task coverage is relatively comprehensive. The nine tasks cover room recognition, visit counting, object deduplication, state changes, spatial topology, and temporal ordering, enabling investigation of the relationship between identity association and subsequent question answering.

3.Data generation has a traceable factual basis. Answers are derived from scene metadata, actual visit sequences, and object state records, with visibility checks and manual review. This makes the labels easier to verify than question–answer pairs written entirely through unrestricted manual annotation.

Critical weaknesses

A.Section 4 and Appendix A.1 describe default sampling at 1 fps, followed by further uniform downsampling when a model’s context limit is exceeded. The appendix lists frame limits of 120, 150, 320, or 480 for some models. Table 2 directly compares their accuracy even though they may receive different amounts of visual evidence. Without comparisons under a common frame budget, it is difficult to distinguish differences in model capability from differences in input information.

B.Section 3 and Appendix Table 7 define T7 as question answering about object changes and location history across sessions. Table 2 reports aggregate T7 accuracy, but the detailed error analysis in Section 4.2 mainly concerns T1 and agent trajectories. T7 results conditional on correct room and object identification are not reported. Since T7 requires identifying the room and object before comparing their states, an incorrect final answer cannot be attributed specifically to state-updating difficulty when the initial identity association is already wrong.

Questions for authors

A.How many frames does each model actually receive, and what proportion of key visit events do these frames cover? Under a common frame budget, do the main rankings in Table 2 and the conclusion that spatially specialized models perform relatively poorly still hold?

B.What proportions of T7 errors arise from incorrect room or object identity association versus incorrect state comparison? How much would T7 accuracy improve if the correct room IDs were provided?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UGea

We thank the reviewer for their constructive feedback and recognition of our well-defined evaluation protocol, comprehensive task coverage, and reliable data generation process.

W. A/Q. A: How many frames does each model actually receive, and what proportion of key visit events do these frames cover? Under a common frame budget, do the main rankings in Table 2 and the conclusion that spatially specialized models perform relatively poorly still hold?

We thank the reviewer for raising this concern. In our original evaluation, videos are sampled at 1 fps, with further downsampling only when required by model-specific context limits. Each room visit includes a 6–9-second look-around within the room, providing approximately 6–9 observations under the full 1-fps setting.

To ensure a fair comparison, we additionally evaluated 11 representative models on ReSpaceBench-mini using the same 128 uniformly sampled frames in Tab. R6. All models use 640 × 480 resolution, except SenseNova-SI-1.5-8B, which uses 420 × 420 to accommodate all 128 frames within its visual token budget. The rank of the models remain generally consistent, with GPT-5.6 Sol and Seed 2.0 Pro retain the top two positions, while the three spatially specialized models remain near the bottom. Although some intermediate rankings change, the results suggest that the relatively poor performance of spatially specialized models cannot be attributed solely to differences frame counts. We will incorporate this controlled comparison into the revised manuscript and clarify the frame-budget and resolution settings in the evaluation protocol.

Table R6: Evaluation under 128-frames on ReSpaceBench-mini

Model1-fps Avg. (%)Rank128-frame Avg. (%)Rank
GPT-5.6 Sol52.22147.561
Seed 2.0 Pro51.56244.892
Seed 2.1 Pro43.11339.114
Qwen3-VL-8B41.11435.565
InternVL3.5-38B39.11*539.333
Gemini 3.1 Pro36.89634.896
GPT-5.6 Luna32.00731.337
GLM-4.6V-Flash-9B29.11*828.4410
SenseNova-SI-1.5-8B29.11*830.228
VST-7B-RL28.67*1029.119
Cambrian-S-7B25.11*1126.6711

* denotes the models using a capped visual input

W.B/Q.B: What proportions of T7 errors arise from incorrect room or object identity association versus incorrect state comparison? How much would T7 accuracy improve if the correct room IDs were provided?

Thank you for your suggestion for this experiment. We agree that T7 errors may arise from incorrect room or object identity association or subsequent state changes. Inspired by your suggestion, in Tab. R7, we conducted an identity-oracle experiment on all 559 T7 questions with strongest evaluated model GPT-5.6 Sol to disentangle these factors, progressively providing ground-truth cues:

  1. Baseline: Original input video.
  2. Room oracle: Each frame includes its room type and unique room ID.
  3. Room + object oracle: Additional to room oracle, this overlays object mask with unique instance IDs. These annotations provide identity correspondence without revealing object states, state changes, or ground-truth answers.

Table R7: T7 Oracle Identity Analysis

Visual InputAccuracyGain vs. Baseline
Original input (baseline)47.0%—
+ Room type and persistent room ID51.9%+4.9
+ Object masks and consistent instance IDs54.0%+7.0

Providing correct room IDs improves accuracy by 4.9, while additional object-level annotations yield a total gain of 7.0. This confirms that identity association contributes to T7 difficulty. These improvements suggest that both room-level and object-level identity association can affect T7 performance, and that explicitly providing identity correspondence helps models better reason about object states across sessions. We will incorporate these findings into the revised paper.

We sincerely thank the reviewer again for their constructive feedback and valuable suggestions. We hope our additional experiments and clarifications have addressed your concerns, and we would be happy to provide further details if needed.

Dear Reviewer UGea,

Thank you again for your detailed and constructive comments. Your insightful suggestions have helped us strengthen the paper and improve the depth and rigor of our empirical analysis. We have updated the paper to incorporate the additional experiments, along with clarifications regarding the evaluation protocol and the sources of spatial memory errors.

We hope these new results address your concerns. We sincerely appreciate your valuable feedback and would be happy to answer any further questions or provide additional clarifications.

Reviewer_cBYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ReSpaceBench, a benchmark specifically for persistent spatial understanding. Their evaluation covers 32 models and 3 general video understanding frameworks, with the best performing model achieves around 50% of the accuracy, compared to human's more than 90%. To improve model's performance, they also present RS-30k. Finetuning qwen3 vl 8b on RS-30k improve models' performance over 25%.

Critical strengths
  1. Large-scale benchmark covering both synthetic case and self-scanned room for the real-world case. This setting covers a diverse set of indoor environments. Annotations are done by human in the real-world examples.
  2. Training set demonstrates the effectiveness of more than 25% of model performance improvement.
Critical weaknesses
  1. My biggest concern is if the benchmark is actually testing spatial memory. Your setting is something like, given the model a video + question, and the question is asking the model to somehow "retrieve" information from the past frames. However, you are inputting the whole video to the model, rather than keep a memory representation / memory state to keep track of what happened before.
  2. The sampling rate is 1fps, would there be a possibility that the action "entering" and "leaving" the room is not presented to model?
Questions for authors
  1. The evaluation setting is a bit unclear. From Figure 1, looks like it's a very long understanding task with the input of traversing 20 rooms together. I'm wondering do you have a subset with less complicated trajectories?
  2. Training is using outcome SFT?
  3. There are many spatial reasoning benchmarks with memory capability (such as your same room identification) in the community, consider comparing them as follows:

[1] Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces, CVPR 2025

[2] SAW-Bench: Learning Situated Awareness in the Real World, ICML 2026

[3] Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs, AAAI 2026

[4] SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence, ICLR 2026

4 comments
AuthorsOpenReview ↗

Response to Reviewer cBYs

We thank the reviewer for their positive feedback and recognition of our benchmark's diverse environments and the effectiveness of RS-30K fine-tuning.

W1: My biggest concern is if the benchmark is actually testing spatial memory...

Thank you for pointing out this distinction. In ReSpaceBench, our spatial memory refers to the ability to associate observations of the same physical places and objects across current and historical visits. Our current evaluation assesses this ability with access to the full visual history. This protocol is consistent with related video-based evaluations such as VSI-Bench and SAW-Bench, while leaving the representation of history open to the evaluated method, similar to OpenEQA.

ReSpaceBench can also support future research in streaming evaluation using the same walkthroughs and annotations. Observations can be presented sequentially, with models updating their memory representations and answering questions from the retained information.

W2: The sampling rate is 1fps, would there be a possibility that the action "entering" and "leaving" the room is not presented to model?

Thank you for raising this concern. At 1 fps, the exact instant of entering or leaving a room may fall between sampled frames, but temporally adjacent observations remain available. Furthermore, each counted visit in our rendered walkthroughs includes an in-room look-around lasting 6–9 seconds, providing approximately 6–9 sampled views under the full 1-fps setting.

The camera trajectories use nominal walking speeds of 0.6–0.8 m/s, motion statistics are summarized in Tab. R5. We will include these statistics and clarify the visit coverage for 1-fps sampled sequence.

Table R5: Walkthrough camera displacement

MetricMeanRange or nominal setting
Walking speed during movement0.662 m/s0.6–0.8 m/s
Average traversal speed, including pauses0.372 m/s0.227–0.503 m/s
camera displacement at 1-fps0.367 m0–0.833 m

Q1: The evaluation setting is a bit unclear... I'm wondering do you have a subset with less complicated trajectories?

Thank you for pointing out. The example in Figure 2 illustrates 20 room visits across 11 distinct rooms, but it is not intended to represent the typical trajectory complexity. To provide a clearer picture, we analyzed the trajectory distributions across the entire ReSpaceBench dataset. The median number of distinct rooms is 6, and 95.3% of inputs involve at most 10 distinct rooms. The median number of room visits, including revisits, is 27, with a median visits-per-room ratio of 4.6. We will include the full-dataset trajectory distributions and performance analysis in the revised manuscript.

Q2: Training is using outcome SFT?

Yes. We use outcome only supervised fine-tuning on RS-30K. Specifically, Qwen3-VL-8B-Instruct is supervised with the training set ground-truth answers, either option letters or numerical values in our case.

Q3: There are many spatial reasoning benchmarks with memory capability (such as your same room identification) in the community, consider comparing them...

Thank you for suggesting these relevant benchmarks. Our current manuscript includes VSI-Bench in Table 1 and cites Seeing from Another Perspective (All-Angles Bench) in the introduction. We will expand the discussion of these works and extend the comparison table to include it alongside SAW-Bench and SpaCE-10, clarifying their task coverage relative to our benchmark.

We sincerely thank the reviewer again for their constructive feedback and thoughtful suggestions, which will help us further improve the clarity and quality of our paper. We hope our responses have addressed your concerns and would be happy to provide any additional details or clarifications if needed.

Dear Reviewer cBYs,

Thank you again for your thoughtful review and encouraging feedback. Your suggestions have been valuable in helping us refine the paper and better communicate our methodology and contributions. Based on your comments, we have updated the manuscript to further clarify our evaluation setting, trajectory complexity, frame-sampling protocol, and comparisons with related benchmarks.

We hope the revised manuscript and our responses have clarified the scope and motivation of our work. We welcome any additional questions or suggestions and would be glad to provide further clarification.

Thank you for your detailed response, my attitude of the paper is towards acceptance. I will maintain my original score.

Thank you for your continued positive assessment of our work. Your thoughtful comments have helped us clarify the scope of our spatial memory evaluation, the frame-sampling protocol, and trajectory complexity, and expand our comparisons with relevant benchmarks that were not fully covered in the original manuscript.

We have incorporated these clarifications into the revised manuscript. We sincerely appreciate your time and constructive engagement throughout the discussion, which have helped improve the clarity and quality of our paper.

Reviewer_Fvy6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary
  • introduces ReSpaceBench, a video-understanding benchmark for persistent spatial memory across revisits to a location in a spatial scene
    • contains 3.2k QAs (218 from real homes) across 9 tasks spanning room association, persistent state tracking, and downstream reasoning
    • a 30k synthetic trainset is also provided (RS-30K), based upon the ProcThor-30K dataset, used to train a Qwen3-VL-8B model from 35.5 to 60.4 on their benchmark
  • QA generation uses 3D scans/simulations to produce a walkthrough with GT info + a video walkthrough
  • 32 MLLMs are evaluated along with 3 agentic frameworks and a group of 3 human evaluators
Critical strengths
  1. it is nice that they include blind and human baselines. the blind eval of the SFT model gives some hope that the model is not simply overfitting to the task.
  2. separating room identity from category, and the subsequent analysis in $\S$4.2 / Fig. 4 is nice
Critical weaknesses
  1. the mini set contains only 50 questions per task, which is too small to draw any meaningful conclusions about a task. yet multiple main conclusions / tables rely entirely on it (eg Tabs. 3, 4, $\S$4.2)
  2. it is unclear how well the benchmark reflects real egocentric video, which is the paper's stated motivation.
  • though 218/3244 questions come from "real homes", it is unclear what this actually means
  • these 218 questions mix FARO scans rendered with BlenderProc (L262-265) and PICO headset recordings (Fig. 3b)
  • transfer is not well-established to real egocentric video, or to other real-world benchmarks
Questions for authors
  1. can you report OOD performance of your SFT model on other benchmarks?it is possible that the model has just been overfit to the task...
    • maybe some similar ones like VSI-Bench/OVO-S
    • and the general benchmarks reported in the original Qwen3-VL paper, to show it has not degraded the performance badly
  2. I am a bit surprised about the SFT models' blind performance. can you report the performance of evaluating the model with shuffled visual inputs (ie a random wrong visual input from the test set)? ensure that the randomly sampled scene is not the same though.

presentation notes / suggestions:

  • fig 1, the house is the most prominent thing in the LHS of the image. but the main point are the challenges (1-3). maybe make those challenges larger / more prominent + legible, or the house smaller? had to zoom in very closely to understand the figure.
  • Tab. 3: results by Task is not meaningful. Probably should aggregate per benchmark taxonomy group to allow for more interpretable comparisons? Should aggregate in Tab 2 also.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Fvy6: Part 1 of 2

We thank the reviewer for the constructive feedback and sincerely appreciate your recognition of the our work and for highlighting the value of our human/blind baselines and room-identity analysis in Section 4.2.

W1: the mini set contains only 50 questions per task, which is too small to draw any meaningful conclusions about a task. yet multiple main conclusions / tables rely entirely on it (eg Tabs. 3, 4, §4.2)
Suggestion 2: Tab. 3: results by Task is not meaningful. Probably should aggregate per benchmark taxonomy group to allow for more interpretable comparisons? Should aggregate in Tab 2 also.

Thank you for the suggestion. Our primary findings regarding the 32 evaluated models and the effectiveness of RS-30K fine-tuning are established on the full 3,244-question benchmark. The room-identity analysis in Section 4.2 uses all 466 T1 questions. Only the human–model comparison, agentic framework evaluation, and frame-sampling ablations are conducted on ReSpaceBench-mini (450 questions) for more manageable evaluation API costs and manual evaluation effort.

Following your suggestion, we have also aggregated results for the tables using Mini-set by the three benchmark taxonomy families in Tables 2 and 3. For Table 3, Association, Persistent State, and Reasoning contain 150, 200, and 100 questions. We reported the per family average in Tab. R1 below. We will update the Tab 2 and 3 in paper accordingly

Table R1: Per-Family Average Performance for Agentic Frameworks. Aggregated scores average their respective task accuracies and overall accuracy is the mean across all nine tasks.

MethodAssociationPersistent StateReasoningOverall
Direct54.748.556.052.2
AVP42.742.549.044.0
DVD37.343.555.044.0
VideoAgent32.726.034.030.0

W2: it is unclear how well the benchmark reflects real egocentric video, which is the paper's stated motivation.

Regarding real-home data. We will explicitly distinguish these two sources in the paper. We clarify that the 218 real-home questions originate from actual furnished residential environments, collected through two pathways:

  • FARO scans: Walkthroughs rendered from scanned real home reconstructed meshes using BlenderProc, preserving real-world geometry and household furnishings. Camera height, field of view, and motion are informed by PICO recordings to approximate headset viewing conditions. An example rendered walkthrough views could be seen in top portion of Figure 2.
  • PICO headset: Direct egocentric recordings, primarily used to capture real-world object-state changes across visits where static scenes cannot capture.

With regarding to data transfer, Appendix A.3 Tab. 6 in the paper evaluates transfer to unseen scene sources, with SFT trained only on synthetic ProcTHOR environments. These gains demonstrate that RS-30K supervision transfers beyond ProcTHOR to unseen scene sources, including real-home-derived data.

Table R2 (Same to Appendix A.3 Table 6): Cross Video Source Transfer

Evaluation SourceBaseSFTGain
3D-FRONT (1,555 QAs)33.255.1+21.9
Real Homes (218 QAs)40.455.2+14.8

Q1: can you report OOD performance of your SFT model on other benchmarks?it is possible that the model has just been overfit to the task...

Thank you for this important suggestion. We agree that improving ReSpaceBench alone does not establish broader generalization. RS-30K-only SFT substantially improves ReSpaceBench but leads to performance degradation on some external benchmarks, highlighting a limitation of training exclusively on task-specific data.

To investigate whether these trade-offs can be mitigated, we additionally conduct mixed-data SFT. Following common spatial-VLM training practices that combine specialized and general instruction data (Daxberger et al., 2025; Cheng et al., 2024). Using the same Qwen3-VL-8B backbone, we compare VLM-3R + LLaVA-Video training with and without RS-30K.

Table R3: Out-of-distribution generalization under different data configurations

BenchmarkOriginal Qwen3-VL-8BVLM-3R + LLaVA-Video dataVLM-3R + LLaVA-Video + RS-30K data
VSI-Bench58.264.366.7
VSI-Bench-Debiased53.255.957.0
Video-MME, 64 frames66.167.067.6
ReSpaceBench35.523.163.6

Adding RS-30K improves VSI-Bench and its debiased subset by 2.4 and 1.1 respectively, alongside a 28.1 gain on ReSpaceBench compared to baseline. Video-MME reaches 67.6, exceeding both the mixture without RS-30K and the original model. These results demonstrate that, when combined with broader training data, RS-30K could provide improved spatial-memory task supervision without sacrificing performance on the evaluated external benchmarks.

AuthorsOpenReview ↗

Response to Reviewer Fvy6: Part 2 of 2

Q2: I am a bit surprised about the SFT models' blind performance. can you report the performance of evaluating the model with shuffled visual inputs (ie a random wrong visual input from the test set)? ensure that the randomly sampled scene is not the same though.

Thank you for suggesting this experiment, we find this an interesting experiment. In fact, during construction, our pipeline includes checks to reduce non-visual shortcuts. These includes balancing answer choices (A/B/C/D) and its semantic label (Bathroom/Bedroom/LivingRoom), constructing paired questions with same question and choices but completely different visual input and answer, and checking for textual artifacts and room-type priors. We repeat these checks after filtering to prevent the selection process from reintroducing biases unintentionally.

We evaluated the same SFT checkpoint on all 3,244 questions under three conditions with matched evaluation settings. For shuffled inputs, videos are randomly sampled from different scenes, excluding questions with identical ground-truth answers.

Table R4: Effect of visual-inputs on on ReSpaceBench

Visual-input conditionNine-task macro accuracy
Correct visual inputs60.4%
Blind29.6%
Randomly mismatched visual inputs20.8%

Removing visual input reduces accuracy by 30.8, while shuffled inputs further reduce it by 8.8. These results indicate that the SFT model relies substantially on visual evidence rather than textual shortcuts. We will include this diagnostic and its protocol in the supplementary material.

Suggestion 1: fig 1, the house is the most prominent thing in the LHS of the image. but the main point is the challenges (1-3). maybe make those challenges larger / more prominent + legible, or the house smaller? had to zoom in very closely to understand the figure.

Thank you for your detailed review of our paper and your thoughtful suggestions. We appreciate your feedback on the presentation of Figure 1. We will revise the figure accordingly, making the three key challenges more prominent and legible while reducing the visual emphasis on the house.

We sincerely thank the reviewer again for their constructive feedback and thoughtful suggestions. We hope our responses have addressed your concerns, and we would be happy to provide further clarification or additional details if any questions remain.

References:

  1. Daxberger et al. MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs. In ICCV 2025.
  2. Cheng et al. SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models. In NeurIPS 2024.

Dear Reviewer Fvy6,

Thank you again for your constructive feedback and valuable suggestions. Your comments have been particularly helpful in identifying areas where we could further improve the paper's experimental validation and presentation. We have updated the manuscript to incorporate the additional experiments and clarifications discussed in our rebuttal.

We hope these revisions satisfactorily address your concerns and make the contributions of our work clearer. We welcome any further questions or suggestions and would be glad to provide additional clarification.

Comments on the paper

2 comments
Area_Chair_1eitOpenReview ↗

Please Engage in the Author-Reviewer Discussion

Dear Reviewers,

The authors have posted their responses to your comments and questions. Please review their replies and share any follow-up comments or questions, especially if any concerns remain unresolved or need further clarification.

Your engagement is important to ensuring a constructive and productive discussion. Thank you for your time and contributions to the ICLR review process.

Best regards,
The AC

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We would like to thank the reviewers for their constructive comments and productive discussion during the rebuttal period. At the post-author-rebuttal phase, the reviewers unanimously recommend acceptance (ratings: 3/3/3). We summarize their recognition of our contributions and our responses to the main concerns below.

Strengths highlighted by the reviewers:

  • Reviewer Fvy6 appreciated experiments including human and blind baselines and the ablation analysis distinguishing physical room identity from semantic room category.
  • Reviewer cBYs highlighted the benchmark’s scale, environmental diversity, and human annotations, together with the effectiveness of RS-30K training data, which improves Qwen3-VL-8B from 35.5% to 60.4%.
  • Reviewer UGea recognized the well-defined revisit protocol, comprehensive coverage of nine tasks, and traceable factual basis ground-truth generation supported by visibility checks and manual review.

Main concerns addressed:

  • Evaluation scope and comparability (Fvy6, UGea): We clarified the evaluation sets used in the experimental analysis (which table uses mini-set), added task-family average to the tables, and evaluated 11 models (including context-constrained models) under a common 128-frame budget, finding that spatially specialized models perform relatively poorly.
  • Memory definition and sampling (cBYs): We clarified the full-history evaluation setting, visit coverage under frame sampling, trajectory complexity, and outcome-only SFT, and expanded comparisons in the manuscript with related benchmarks.
  • Generalization and visual grounding (Fvy6): We clarified the two real-home data sources, reported cross-source transfer, and added mixed-data SFT experiments showing gains on three external benchmarks. We also conducted the reviewer suggested Blind and mismatched-video input controls further support reliance on visual evidence.
  • Task T7 Further Analysis (UGea): Following the reviewer suggestions, we also added identity-oracle experiments on all 559 T7 questions. Room identity cues improve accuracy from 47.0% to 51.9%, and additional object indentity raise it to 54.0%, showing the benefit of identity information.

These experiments and clarifications have been incorporated into the revised manuscript. We sincerely thank the reviewers again for their thoughtful feedback and constructive suggestions, which have helped us improve the clarity and rigor of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.