acceptodds
Under review as a conference paper at ICLR 2027

MUSHROOM: Benchmarking Long-Term Spatial Reasoning in Multi-Room Environments

Abstract

Long-term spatial reasoning requires vision-language models (VLMs) to consolidate partial observations into spatial knowledge and update it as spatial states change. We introduce MUSHROOM, a benchmark for long-term spatial reasoning in multi-room environments, comprising 230 egocentric videos that traverse multiple rooms, where room boundaries disrupt spatial continuity. We design tasks along two complementary axes: Static Spatial Reasoning within a stationary configuration, and Dynamic Spatial Reasoning over state changes. We further propose SPORE, a training-free pipeline that organizes a single video into a structured representation: an object-centric spatial graph. Our evaluation demonstrates that contemporary VLMs have fundamental limitations in cross-region reasoning and updating spatial states, while SPORE substantially improves both static and dynamic reasoning, with consistent gains on real-world indoor benchmarks. Further analysis shows that the spatial graph links temporally distant observations and stabilizes geometric estimation, yet cross-region reasoning falters under frequent transitions, a pattern reminiscent of the doorway effect.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Video Understanding, Long-Term Spatial Memory, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RhdpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MUSHROOM (a benchmark for spatial reasoning in multi-room videos) and SPORE (a training-free method that uses object-centric spatial graphs to support vision-language models).

Critical strengths

The paper extends spatial-reasoning research to multi-room exploration. Two critical strengths are listed below.

  1. The benchmark repeats the same camera moving trajectory before and after object relocation, and then evaluates both static relations and object states. This provides a useful experiment setting that helps determine whether the model can memorize and distinguish earlier object states and update them after objects are moved across rooms.

  2. The empirical comparisons among video-only inputs, predicted graphs, and ground-truth graphs reveal useful task-specific differences. This provides more specific evidence about which weakness can be improved by providing information on object positions and room assignments.

Critical weaknesses

Three main weaknesses are shown below. Apart from them, I also have a “meta” concern on this benchmark research. For easy reference, I mark it as 4.

  1. Some of the reported performance improvements are not clearly explained. Appendix C.1 fixes 64 uniformly sampled frames as the VLM input for all models, whereas Section 4, Algorithm 1 and Appendix B.1 indicate that geometry, detection and tracking run over the full frame sequence and captioning uses 16 observations per node, so SPORE observes far more frames than the video-only baselines; the paper should state this asymmetry explicitly and report the total frame count per video. The paper also does not report how detection, tracking, and merging influence the final question-answering performance, since Table 6 and Appendix B.3 report only graph quality.

  2. The distinction between earlier and updated object states is still not fully explained. How the models use timestamps is not clear. The code retains frame indices, but their use in the final question-answering input remains unclear, particularly when only the predicted graph is provided. This leaves a gap in understanding how the method supports relocation recall and direction questions.

  3. The error analysis does not decouple the different factors affecting the results. Appendices D.3 and D.4 provide recall comparisons across models, clustering of frame embeddings by region, and attention analysis, and the doorway-effect analysis in Section 5.3 (Figure 5c) pools T2–T4 without controlling for differences in task composition or path complexity across the room-transition-density quintiles. Prior and updated states also remain tied to the first and second traversals. These observations therefore do not sufficiently establish why cross-room reasoning and state updating remain difficult.

  4. Given the training-free design of SPORE, the benchmark would be more informative if the paper could distinguish between “what can be solved through explicit computation and state retrieval without additional training” and “what difficulties remain and might benefit from further training”. Appendix B.2 already scores the predicted MUSHROOM nodes geometrically and Appendix B.4 answers VSI-Bench questions directly from node coordinates. Reporting the corresponding direct-computation accuracy on the MUSHROOM tasks (T2, T3, T7, T8) from the same predicted nodes while keeping the perception outputs fixed would help distinguish coordinate calculation and state retrieval from remaining perception or state-estimation errors, and clarify where additional training might help.

Questions for authors

There are 5 questions about the research details.

  1. Please confirm that geometry, detection and tracking run over the full frame sequence while the VLM baselines receive 64 frames, and report the number of frames used at each stage, including the 16 observations per node used for captioning.

  2. Could you provide the complete model input for one relocation-recall or relocation-direction question using only the predicted graph, showing how the two candidate locations are presented and how their temporal order can be determined?

  3. Appendix B.4 already evaluates answers computed from node coordinates on VSI-Bench. Could you report a similar comparison on MUSHROOM, using the same predicted nodes to compare direct coordinate calculations or time-based state lookup with VLM answers?

  4. For relocation questions that require both states, do the sampled frames provide sufficient visual evidence before and after relocation? Please describe how visibility was checked separately for the two traversals.

  5. Does accuracy still decrease with room-transition density when comparisons are made within the same task (T2, T3 and T4 separately) and the same scene, given that the density metric is already normalised by the frame gap between the relevant observations?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer Rhdp (1/4)

We are grateful to Reviewer Rhdp for the detailed and insightful comments. We are pleased that you found our before-and-after traversal design useful, and that the comparison across video-only inputs, predicted graphs, and ground-truth graphs offers task-specific evidence on where structured representations help.

For weaknesses and questions, we address each point below and will incorporate the clarifications and additional analyses into the revised manuscript.


Response to Weaknesses:

W1. Input Asymmetry and Stage-wise Contribution to QA

We thank the reviewer for pointing out this important point. In the revised manuscript, we will clarify in the experimental setup that SPORE constructs spatial graphs from the full frame sequence, and report frame statistics per video.

  • Frame Usage across Stages: Each video in MUSHROOM contains 839.3 frames on average (range: 586–1,233). SPORE processes all frames for geometry estimation, detection, and tracking. For captioning, it picks up to 16 observations per node, the most confident one in each time interval, and gives the captioner four of them as full frames with close-up crops of the two largest boxes (Appendix B.1).

  • Stage-wise Contribution to MUSHROOM (QA Task): To examine how each stage influences the final performance, we provide Qwen3-VL-8B with spatial graphs constructed at each stage of Table 6 and evaluate it on MUSHROOM. Values in parentheses denote gains over video-only inputs with 64 frames. Graphs from each stage progressively improve performance over video-only inputs, with merging particularly benefiting dynamic reasoning, where the MRD gain grows from 4.50%p after tracking to 10.94%p.

    StageMRSMRDOverall
    Detection only44.13 (+10.14)37.95 (+2.30)40.44 (+5.46)
    + Tracking45.36 (+11.37)40.15 (+4.50)42.25 (+7.27)
    + Merging (SPORE)46.39 (+12.40)46.59 (+10.94)46.51 (+11.53)

We believe the suggested stage-wise analysis offers further insight into how each component of SPORE contributes to long-term spatial reasoning.

W2. Distinguishing Prior and Updated States

We appreciate this question, which allows us to clarify how SPORE supports reasoning about state changes.

  • Representing State Changes without Frame Indices: SPORE retains frame indices during graph construction but excludes them from the final spatial graph. For relocated objects, the graph preserves both spatial states without indicating their temporal order. This design lets the graph provide spatial information while leaving temporal ordering to be inferred from visual observations.
  • Role of Visual Observations on Dynamic Spatial Reasoning: Since the graph alone gives no temporal cue, the VLM grounds each state in the video to determine their order, reaching 46.59% on MRD with Qwen3-VL-8B (+10.94%p over video-only inputs; see W1). We will clarify this in the revised manuscript and provide an example input (see Q2).
AuthorsOpenReview ↗

Responses to Reviewer Rhdp (2/4)

W3. Decoupling Factors in the Error Analysis

We are grateful for this detailed suggestion, which helps us further substantiate why cross-region reasoning and state updating remain difficult. We address the two factors raised below.

  • Decoupling Prior and Updated States from Temporal Position: Although prior and updated states are tied to the first and second traversals, Figure 4(b) shows that video-only recall declines with later observation positions even within the same traversal. This indicates that the higher recall of prior states (Appendix D.3) reflects their earlier temporal position rather than a preference for prior states. We will clarify this distinction in the revised manuscript.

  • Doorway Effect within Tasks: To control for task composition, we separately analyze T2, T3, and T4 across the room-transition-density quintiles. Results are reported for Qwen3-VL-8B with video-only inputs, from the 1st quintile (lowest density) to the 5th quintile (highest density). Performance degrades with room-transition density in every task, indicating that the doorway effect consistently appears across tasks (see Q5 for further control of scene-level differences).

    TaskQ1Q2Q3Q4Q5Δ (Q1→Q5)
    T2: Relative Distance42.0332.6126.0920.2916.67−25.36
    T3: Relative Direction48.6238.1827.2722.0211.01−37.61
    T4: Static Pathfinding31.9632.9930.9321.8810.31−21.65
    Overall41.2834.4927.8321.2813.08−28.20

W4. Explicit Computation vs. Remaining Difficulties

We thank the reviewer for this constructive suggestion, which helps distinguish what can be resolved through explicit computation from what remains for further training.

  • Experimental Setup: Following Appendix B.4, we compute answers to T2, T3, T7, and T8 directly from the same predicted nodes without a VLM, keeping all perception outputs fixed. For T7 and T8, we determine the temporal order of the two retrieved states using the frame indices retained during graph construction. Answer options whose referenced objects are not retrieved are excluded, and questions whose queried object is not retrieved are counted as incorrect.

    MethodT2: Relative DistanceT3: Relative DirectionT7: Dynamic DistanceT8: Dynamic Direction
    Direct Computation68.2672.3948.1945.80
    SPORE (Qwen3-VL-8B)51.8837.2947.0633.28
    SPORE (Cambrian-S-7B)36.9636.7535.0732.82
  • Direction Is Limited by VLM Reasoning: On T3, direct computation reaches 72.39%, whereas Qwen3-VL-8B reaches 37.29% from the same nodes (−35.10%p). This suggests that the VLM struggles to transform world coordinates into the frame of the queried viewpoint, which could be improved through training on reasoning over structured representations.

  • Relocation Is Limited by Retrieving Both States: Direct computation drops by over 20%p on T7 and T8 compared to the static tasks, while Qwen3-VL-8B is only 1.13%p below it on T7. Since a relocated object must be retrieved at both its prior and updated positions, only 41.8% of relocated objects are covered, compared to 74.8% of static objects (Table 7). T8 combines both limitations, making it the most challenging of the four tasks (33.28%).

We believe this distinction strengthens the diagnostic value of MUSHROOM and points to training for reasoning over structured representations as a promising future direction. We will include this analysis in the revised manuscript.

AuthorsOpenReview ↗

Responses to Reviewer Rhdp (3/4)

Response to Questions:

Q1. Frame Counts at Each Stage

As detailed in W1, SPORE processes the full frame sequence while the VLM baselines receive 64 frames. To disentangle the effect of frame usage, we additionally construct spatial graphs from the same 64 frames given to the VLM, as well as from 128, 256, and 512 frames. Values in parentheses denote gains over video-only inputs with Qwen3-VL-8B (Table 1).

  • Gains Beyond Frame Usage: Even with identical frames, SPORE improves over video-only inputs by +8.05%p, indicating that the gains stem from the structured representation rather than additional frames. Using more frames further improves performance, as denser observations yield more complete graphs.

    FrameMRSMRDOverall
    6443.14 (+9.15)42.95 (+7.30)43.03 (+8.05)
    12843.94 (+9.95)43.30 (+7.65)43.56 (+8.58)
    25645.40 (+11.41)45.34 (+9.69)45.37 (+10.39)
    51245.78 (+11.79)45.28 (+9.63)45.48 (+10.50)

Q2. Example Input with the Predicted Graph

We thank the reviewer for requesting a concrete example, which helps clarify how relocated objects are presented to the VLM. We provide below the predicted graph input when “white table lamp” is the queried object in dynamic tasks, where it has been relocated in the video.

  • Node Retrieval: Following Algorithm 1, SPORE returns up to two nodes for the queried object, where the second lies farther than δ from the first (Appendix B.1). Accordingly, the relocated object is retrieved as two nodes corresponding to its two spatial states, which are provided under the query object in JSON format.
  • Excluded Temporal Information: SPORE retains the frame indices and bounding boxes of each node’s observations during graph construction, but excludes them from the input. Node id is assigned after merging, grouped by category and ordered by the number of detections, and does not reflect temporal order.
"white table lamp": [
  {
    "id": 155,
    "name": "lamp",
    "coarse_vote": {"lamp": 1.0},
    "center": [0.158, 0.026, 0.101],
    "caption": "The object is a table lamp with a dark gray shade, ..., it sits on a light-colored wooden bedside table positioned next to the bed, ...",
    "room": "bedroom"
  },
  {
    "id": 151,
    "name": "lamp",
    "coarse_vote": {"lamp": 1.0},
    "center": [0.499, 0.030, -0.122],
    "caption": "The masked object is a white table lamp with a pleated, ..., sitting on a white circular dining table in the center of a modern living room. ...",
    "room": "living room"
  }
]

  • Determining Temporal Order: With only the predicted graph, the two candidate states carry no temporal cue, so their order cannot be determined from the graph alone. As described in W2, the VLM determines the order by grounding each state in the video.

Q3. Direct Computation on MUSHROOM

We thank the reviewer for proposing this comparison. Please refer to W4, where we compute answers to T2, T3, T7, and T8 directly from the same predicted nodes, using the retained frame indices for state lookup in T7 and T8.

Reviewer_PaGzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MUSHROOM, a benchmark designed to evaluate vision-language models (VLMs) on long-term static and dynamic spatial reasoning within multi-room environments featuring object relocations. To alleviate the failure of contemporary VLMs in maintaining spatial continuity across room transitions and tracking state updates, the authors also propose SPORE, a training-free pipeline that abstracts egocentric videos into object-centric 3D spatial graphs. Comprehensive evaluations across proprietary, general, and spatial VLMs demonstrate that current models struggle with cross-region reasoning and state tracking, whereas structured spatial representations yield notable improvements.

Critical strengths

Addressing a critical benchmark gap with dynamic multi-room environments: Most existing 3D/video spatial reasoning benchmarks evaluate models strictly within static configurations. MUSHROOM introduces a multi-traversal setup capturing explicit object relocations across room boundaries, filling an important and underexplored niche by systematically testing models' ability to maintain and update spatial memory over long-horizon exploration.

Critical weaknesses

Lack of evaluation on the most recent models to provide an up-to-date snapshot: For a benchmark contribution, providing a timely and representative snapshot of the current state-of-the-art capabilities is crucial. The current evaluation omits recent frontier open-source and proprietary models, which diminishes the benchmark's immediate diagnostic value and reference utility for the broader community.

Imprecise motivation and overclaiming regarding multi-room traversal: Traversing across multiple rooms is not unique to this work; several existing long-video spatial intelligence benchmarks (e.g., MMSI-Video-Bench) already feature indoor trajectories spanning multiple rooms. The paper's core and most prominent contribution is benchmarking dynamic spatial state updates rather than multi-room navigation per se, yet the current narrative in the introduction conflates these aspects and does not clearly delineate its positioning against existing multi-room video benchmarks.

Questions for authors

Could the authors refine the motivation to clearly distinguish MUSHROOM from existing long-video benchmarks that already include multi-room navigation (such as MMSI-Video-Bench), highlighting that the true differentiator lies in dynamic state updating and re-traversal?

Could the authors evaluate and report results on more recent frontier proprietary and open-source models to provide the community with a timely snapshot of contemporary VLM capabilities on MUSHROOM?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer PaGz

We thank Reviewer PaGz for the careful review and constructive suggestions. We are encouraged that you see MUSHROOM as filling an underexplored niche, particularly in testing whether models can maintain and update spatial memory as objects move across rooms.

For weaknesses and questions, we address each point below and will incorporate the clarifications and additional analyses into the revised manuscript.


Response to Weaknesses & Questions:

W1&Q2. Evaluation on Recent Frontier Models

We appreciate the reviewer’s emphasis on providing a timely snapshot of contemporary VLM capabilities. To this end, we additionally evaluate four recent frontier models that natively accept video input, covering both proprietary and open-source families: Gemini 4 Argon, Qwen3.8-27B, GLM-5.3-Flash, and MiMo-V2.6-Flash. All models are evaluated under the same 64-frame setting as in Table 1, and the results are shown in the table below.

  • The recent models outperform the best model in Table 1 (34.98%) but all remain below 42%. This indicates that long-term spatial reasoning across multiple regions remains challenging for current VLMs.

  • Even the strongest, Gemini 4 Argon (41.62%), falls short of Qwen3-VL-8B with SPORE (46.51%). Moreover, SPORE is training-free, so it can be applied directly to these models.

    ModelMRS (T1-T4)MRD (T5-T9)Overall
    Gemini 4 Argon41.4041.7741.62
    GLM-5.3-Flash37.4440.4339.22
    MiMo-V2.6-Flash35.6037.9837.02
    Qwen3.8-27B33.7636.3235.29

As requested in the second question, we will include these results in the revised manuscript, so that MUSHROOM serves as an up-to-date reference for tracking progress in long-term spatial reasoning.

W2&Q1. Positioning Against Existing Multi-Room Video Benchmarks

We thank the reviewer for recognizing dynamic spatial state updating as a core contribution and for pointing us to MMSI-Video-Bench [1]. In MUSHROOM, multi-room traversal serves as a means to evaluate long-term spatial reasoning that relates objects observed in different regions at temporally distant moments. We view this focus as complementary to existing multi-room video benchmarks and clarify the distinctions below.

  • Cross-Region Integration by Design: Every static question relates specific objects observed separately in different regions, with the reference and all answer options placed in mutually distinct regions (Appendix A.2). This requires VLMs to link temporally distant observations and integrate separately observed regions into a single layout.
  • Region Transitions as a Core Element:
    • Continuous Observation of Transitions: Our videos are continuous (Appendix A.1), so every region transition is observed and the viewer’s position and heading remain well defined across regions. In contrast, many multi-region questions in [1] are drawn from web house-tour videos, which are often discontinuous across regions with cuts or fade transitions.
    • Transition-Grounded Analysis: Annotated region identities and transitions allow us to diagnose why long-term spatial reasoning fails beyond temporal distance. Room transition density further shows that performance declines as region boundaries are crossed more frequently, a pattern reminiscent of the doorway effect (Figure 5(c)).
  • Starting Point for Spatial Representations: Beyond the benchmark, our oracle analysis shows that structured representations substantially narrow the gap in long-term spatial reasoning (Section 5.2). SPORE provides a practical starting point by constructing such representations from a single RGB video without training or calibration (Section 4).

As suggested in the first question, we will refine the motivation with an emphasis on dynamic spatial state updates, articulate the challenges of multi-region exploration, including spatiotemporal relations between objects across regions and region transitions, and clarify our positioning against existing multi-room video benchmarks.


[1] Lin et al. MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence. arXiv, 2025.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer PaGz,

Thank you for your time and effort in reviewing our paper. We sincerely appreciate your valuable feedback and have carefully considered all the points you raised.

We have posted our response to your initial review and revised the manuscript accordingly. Following your suggestions, we made several revisions, including a clarified positioning against existing multi-room video benchmarks (Section 1) and an evaluation of recent frontier VLMs (Appendix F.1), along with other clarifications throughout the paper.

As the discussion period is coming to a close, we wanted to reach out to see whether our responses have sufficiently addressed your concerns. We are happy to discuss any aspects of the paper that require further clarification—please let us know if there are any points you would like us to elaborate on.

Once again, we greatly appreciate your thoughtful review and insights.

Best regards,

The Authors

Thank you to the authors for their detailed response. Taking into account the overall contributions of the paper, I will keep my current rating.

AuthorsOpenReview ↗

Thank you Reviewer PaGz

Dear Reviewer PaGz,

Thank you for taking the time to read our rebuttal and for your thoughtful response.

We truly appreciate your constructive feedback, which gave us the opportunity to better clarify our core contributions.

Thank you again for your time and effort.

Best regards,

The Authors

Reviewer_jkysOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark designed to evaluate long-term spatial reasoning and spatial memory updating for Vision-Language Models (VLMs) in multi-room environments. It consists of 230 egocentric videos rendered in NVIDIA Isaac Sim across 25 synthetic indoor scenes to enable testing foundational models on both static layout and dynamic conditions (e.g., when they relocate some objects across the rooms). Additionally, the authors propose SPORE, as a training-free pipeline that extracts 3D object-centric spatial nodes from single videos using off-the-shelf geometry, detection, tracking, and captioning modules to improve cross-region reasoning. While the problem of spatial knowledge consolidation across regions to enhance the overall quality VLM-guided navigation and scene understanding is timely and relevant, the technical novelty and empirical depth of the are somewhat limited.

Critical strengths

• The problem of long-term spatial reasoning and state tracking across multi-room environments is timely and relevant to embodied AI and video understanding.

• The paper is well-written, logically structured, and clearly articulates its motivation and experimental findings.

• Testing models on both static configurations and dynamic object relocations provides a clear look at how spatial memory degrades over successive visits.

• SPORE consistently improves the performance of models like Qwen3-VL and Cambrian-S over pure-video baselines on both MUSHROOM and existing indoor benchmarks like VSI-Bench and ReVSI (Tables 1–3).

Critical weaknesses

• The dataset is rendered entirely within NVIDIA Isaac Sim using some synthetic scenes, twice-scale clearance, and pre-scripted trajectories. This approach fail to mimic the fine resolution and realistic complexities of real-world images. For instance, complex backgrounds, sensor noise, dynamic lighting, camera shake and motion artifacts, and clutter typical of real-world embodied navigation can substantially influence the results. Consequently, it is unclear how well the findings transfer to real-world robotic settings.

• The benchmark evaluates spatial reasoning purely through 5-choice text-based MCQs (e.g., discrete action sequences like "turn left, go straight") rather than closed-loop execution, continuous coordinate estimation, or active exploration. This reduces a complex embodied navigation and interaction challenge to offline video QA, which is susceptible to language priors and heuristic option elimination.

• The novelty of paper is not well established and I would expected to see some theoretical foundations, analysis, or deep investigation of developed concepts. It includes testing of multiple VLAs for simple scenarios with simplistic tests (such as MCQ, VQA,…) …, which offers limited insight. Inclduing comparison with trajectory prestation using VLAs could have also improved the paper.

• Because SPORE relies on cascading independent modules, errors compound throughout the pipeline. As shown in Table 7, replacing predicted poses and boxes with ground truth yields massive jumps in matching accuracy (from 68.4% to 82.7%), underscoring how heavily performance is bottlenecked by the off-the-shelf perception stack rather than the representation itself.

Questions for authors

1- Please elaborate on key contributions and the novelty of the paper. It is understandable that including more contextual information, such as node description will improve the accuracy of VLA interpretation compared to pure video. Can you elaborate on fundamental limits (e.g., information-theoretic limits) of the reported gain. How does it relate to the overlap and correlation models across the regions.
2- One of main concerns is the method of evaluations. The method is evaluated with passive multiple-choice text options like "turn left, go straight". Why not using actual navigation success and trajectory error directly in Isaac Sim? Have you tested whether SPORE can control an embodied agent or VLA model rather than just feeding prompt tokens to a VLM? 3- In Table 7, giving the pipeline ground-truth bounding boxes and camera poses substantially improves the overall and relocation matching. Since performance depends so heavily on the off-the-shelf detector and SLAM tools, how do you show that SPORE is a new representation rather than just a pipeline of existing models? How does it handle real-world issues like motion blur and drift? 4- The benchmark uses only 25 clean synthetic scenes with static lighting, flat floors, and double-scale robot clearance. Beyond the zero-shot coordinate tests on VSI-Bench, what evidence is there that tracking moving objects across rooms actually works in noisy, cluttered real-world environments? In general, have you tried testing in realistic conditions.

6 comments
AuthorsOpenReview ↗

Responses to Reviewer jkys (1/4)

We sincerely thank Reviewer jkys for the thoughtful feedback and for recognizing the timeliness of long-term spatial reasoning in multi-room environments. We appreciate the acknowledgment that our benchmark reveals how spatial memory degrades over successive visits, and that SPORE consistently improves over video-only baselines.

For weaknesses and questions, we address each point below and will incorporate the clarifications and additional analyses into the revised manuscript.


Response to Weaknesses:

W1. Transferability to Real-World Environments

We thank the reviewer for raising this important concern. We acknowledge that synthetic environments do not fully capture the visual complexity of real-world recordings, as discussed in Appendix F.3. We clarify our benchmark motivation and provide additional analyses to address the concern about real-world applicability.

  • Controlled Setting for Benchmarking Long-Term Spatial Memory: To benchmark VLMs' ability to consolidate partial observations into a coherent global spatial representation and update it as the environment changes, we deliberately designed MUSHROOM with controlled visual conditions to minimize confounding perceptual factors. Even under these conditions, VLMs struggle with both spatial consolidation and state updating (Table 1).

  • Robustness to Real-World Complexities: To assess whether SPORE remains effective under realistic visual conditions, we analyze how video characteristics affect performance on real-world indoor videos from VSI-Bench [1] and ReVSI [2]. Using Qwen3-VL-8B, we split the videos at the median of each characteristic and report overall accuracy (%) on the same MCQ tasks as in Table 2 (Qwen3-VL-8B: 48.03, SPORE: 52.32).

    • Real-World Video Characteristics: The real-world videos are handheld recordings captured while moving through indoor environments. We measure three characteristics to quantify their visual conditions:
      • Image Sharpness: Variance of the Laplacian, where lower values indicate reduced image sharpness.
      • Camera Jitter: Root mean square (RMS) deviation of the estimated camera trajectory from its smoothed trajectory, measuring irregular camera motion.
      • Camera Speed: Median frame-to-frame global image displacement per second (px/s), measuring the magnitude of camera motion.
    • Result: As shown in the table below, although these videos exhibit varying image sharpness, non-negligible camera motion and instability beyond our controlled synthetic setting, SPORE consistently outperforms the video-only baseline across all conditions. This indicates that SPORE remains robust to realistic visual complexity.
    Visual Condition (Median)LevelQwen3-VL-8BSPOREΔ (%p)
    Image Sharpness (93.93)Low (Blurry)45.3850.28+4.90
    High (Sharp)49.6953.60+3.91
    Camera Jitter (8.45)Low (Stable)49.7856.67+6.89
    High (Shaky)46.4148.28+1.87
    Camera Speed (247.53)Low (Slow)50.6552.26+1.61
    High (Fast)45.1952.39+7.20

W2. MCQ-Based Evaluation

We thank the reviewer for this thoughtful comment. Through language-based MCQs, MUSHROOM evaluates the spatial reasoning capability of VLMs as the core reasoning component of embodied agents. To complement the MCQ format, we further analyze continuous geometric estimation (Figure 5(a)–(b)).

  • Language-Based Reasoning as a Foundation for Embodied Agents: Recent VLAs ground their actions in language-based reasoning, such as subtask descriptions or chain-of-thought [3, 4], indicating that linguistic competence in spatial reasoning underlies embodied intelligence. In this light, MUSHROOM adopts language as an anchor to evaluate how VLMs consolidate, update, and express spatial knowledge.
  • Geometric Estimation beyond Text-Based MCQs: To complement language-based evaluation and mitigate concerns about language priors and option elimination, Section 5.3 provides the analysis of geometric estimation through mean absolute distance error and mean angular error. With video-only inputs, distance errors grow with the true distance and angular errors remain large, whereas SPORE substantially reduces both (Figure 5(a)–(b)).

Beyond offline Video QA, we evaluate closed-loop navigation in the same environments; please refer to Q2 for the setup and results.

AuthorsOpenReview ↗

Responses to Reviewer jkys (2/4)

W3. Novelty and Depth of Analysis

We appreciate the reviewer’s feedback on the depth of our analysis. Grounded in object–region association [5], MUSHROOM captures the spatiotemporal structure of multi-region exploration, where continuous trajectories connect separately observed regions through region transitions. Building on this, we analyze long-term spatial reasoning along temporal and cross-region dimensions, verifying the findings through controlled analysis.

  • Difficulties in Long-Term Spatial Reasoning:

    • Object–Region Assignment as a Core Component: Our ablation study on the oracle graph shows that object–region assignment is a core component of long-term spatial reasoning, as dynamic spatial reasoning degrades without it even with exact object coordinates (Table 3).
    • Temporal Integration: Video-only errors increase sharply as relevant observations become temporally distant, and recall of spatial states declines as they are observed later in the exploration (Figure 4).
    • Cross-Region Integration: Performance degrades as room transitions become more frequent, a pattern reminiscent of the doorway effect in human cognition, where crossing a doorway impairs memory of the previous room (Figure 5(c)).
  • Controlled Analysis of Region Transitions: For a deeper investigation of this effect, we isolate the effect of region transitions within the same scene by assigning room-transition-density quintiles (Section 5.3) within each scene for T2–T4, using Qwen3-VL-8B with video-only inputs (64 frames). Performance still declines from the 1st to the 5th quintile in every task, with an overall drop of 20.64%p, clarifying the effect of region transitions independent of scene-level differences.

    TaskQ1Q2Q3Q4Q5Δ (Q1→Q5)
    T2: Relative Distance36.2339.8628.9918.1214.49−21.74
    T3: Relative Direction38.5331.8231.8227.5217.43−21.10
    T4: Static Pathfinding32.9935.0523.7121.8814.43−18.56
    Overall36.0535.9428.4122.1615.41−20.64
  • Trajectory-Level Evaluation (Open-Loop): Following the reviewer’s suggestion, we evaluate free-form navigation instructions generated by Qwen3-VL-8B for T4 and T9 without answer options, through floor-plan execution (Appendix A.2). Both GT graphs and SPORE improve success rates, but performance remains limited, suggesting the need for explicit navigational topology (e.g., room connectivity and traversable paths). We further evaluate closed-loop navigation with an embodied agent in Q2.

    InputT4 (Success Rate: %)T9 (Success Rate: %)
    Baseline (Video Only)11.784.74
    Video + GT graph21.0712.41
    SPORE18.396.20

W4. Dependence on Off-the-Shelf Modules

We thank the reviewer for this point. We respectfully argue that SPORE goes beyond a simple cascade of off-the-shelf modules, as each component resolves a practical constraint of constructing spatial graphs from a single RGB video without privileged geometric or semantic annotations.

  • Resolving Practical Constraints: SPORE estimates camera parameters without calibration, detects objects with open-vocabulary labels, and merges fragmented tracklets across intermittent observations into consistent nodes (Section 4; Appendix B.1).

    • Setup: To examine the contribution of each step, we provide Qwen3-VL-8B with spatial graphs constructed at each stage of Table 6 and evaluate it on MUSHROOM. Values in parentheses denote gains over video-only inputs with 64 frames.
    • Result: When provided to the model, graphs from each stage progressively improve performance, with merging particularly benefiting dynamic reasoning (MRD: +4.50%p → +10.94%p). This suggests that consolidating fragmented observations into consistent nodes helps represent each spatial state of relocated objects.
    StageMRSMRDOverall
    Detection only44.13 (+10.14)37.95 (+2.30)40.44 (+5.46)
    + Tracking45.36 (+11.37)40.15 (+4.50)42.25 (+7.27)
    + Merging (SPORE)46.39 (+12.40)46.59 (+10.94)46.51 (+11.53)
  • Upper Bound with Ground-Truth Components: Table 7 replaces predicted components with ground truth to estimate how much SPORE can gain as perception modules advance, rather than to expose a limitation of the representation. Even with entirely predicted components, SPORE improves Qwen3-VL-8B by 11.53%p over video-only inputs (Table 1).

  • Robustness to Module Choices: Different detectors yield comparable matching accuracy (OWLv2: 68.4%, YOLO-World: 67.1%, DART: 68.8%; Table 8), whereas removing captions reduces matching by 25.3%p (Table 10), indicating that performance depends more on what the representation encodes than on a specific module.

AuthorsOpenReview ↗

Responses to Reviewer jkys (3/4)

Response to Questions:

Q1. Key Contributions and Limits of the Gain

We appreciate this insightful question. For our contributions and novelty, please refer to W3; here, we focus on the limits of the reported gains. Regarding the overlap and correlation across regions, we empirically analyze how the gains depend on the fragmentation of observations by region transitions, defining density quintiles within each scene to account for scene-level differences.

  • Performance Gains under Cross-Region Fragmentation: To examine cross-region integration independently of state changes, we analyze the static cross-region tasks (T2–T4), as dynamic tasks are also influenced by the temporal position of observations (Figure 4(b)).

    • Setup: We measure room-transition density as the number of region transitions divided by the frame gap between relevant observations and define density quintiles within each scene. This allows us to examine the effect of region transitions while accounting for scene-level differences in density distributions.
    • Result: As density increases, video-only accuracy drops sharply (−20.64%p), whereas SPORE shows a smaller decline (−7.56%p), with its gain increasing from 6.68%p to 19.76%p. These results provide empirical evidence consistent with explicit spatial representations preserving spatial information across fragmented observations, with increasing benefits as region transitions become more frequent.
    InputQ1Q2Q3Q4Q5Δ (Q1→Q5)
    Video Only (64 frames)36.0535.9428.4122.1615.41−20.64
    SPORE42.7343.1941.1639.0735.17−7.56
    Gain+6.68+7.25+12.75+16.91+19.76

Q2. Closed-Loop Navigation Evaluation

We thank the reviewer for this suggestion. Beyond the open-loop trajectory-level evaluation in W3, we further conduct closed-loop navigation with OneVLA [4] in the same environments without additional training, examining whether SPORE can support an embodied agent.

  • Setup: We sample 100 questions each from T4 and T9. OneVLA observes the same video and is then placed at the starting position to navigate to the target.

  • Metric: We report the oracle success rate (OSR), a standard vision-and-language navigation metric that measures target reachability regardless of where the agent stops. An episode succeeds if the agent comes within 3 m of the target at any point along its trajectory.

  • Result: As shown in the table below, although SPORE improves OSR on both tasks, success remains limited, particularly on T9, which requires recalling the original location of a relocated object. This highlights the need to properly evaluate long-term spatial reasoning in multi-region environments, which underlies navigation execution.

    InputT4: Static PathfindingT9: Dynamic Pathfinding
    Video Only (64 frames)17.00.0
    SPORE24.02.0

As MUSHROOM evaluates video spatial reasoning in VLMs and SPORE supports it with a spatial representation, closed-loop evaluation and control with embodied agents or VLA models fall outside the main scope of this work; we view this experiment as an initial exploration and leave these directions to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.