- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MUSHROOM (a benchmark for spatial reasoning in multi-room videos) and SPORE (a training-free method that uses object-centric spatial graphs to support vision-language models).
Critical strengths
The paper extends spatial-reasoning research to multi-room exploration. Two critical strengths are listed below.
-
The benchmark repeats the same camera moving trajectory before and after object relocation, and then evaluates both static relations and object states. This provides a useful experiment setting that helps determine whether the model can memorize and distinguish earlier object states and update them after objects are moved across rooms.
-
The empirical comparisons among video-only inputs, predicted graphs, and ground-truth graphs reveal useful task-specific differences. This provides more specific evidence about which weakness can be improved by providing information on object positions and room assignments.
Critical weaknesses
Three main weaknesses are shown below. Apart from them, I also have a “meta” concern on this benchmark research. For easy reference, I mark it as 4.
-
Some of the reported performance improvements are not clearly explained. Appendix C.1 fixes 64 uniformly sampled frames as the VLM input for all models, whereas Section 4, Algorithm 1 and Appendix B.1 indicate that geometry, detection and tracking run over the full frame sequence and captioning uses 16 observations per node, so SPORE observes far more frames than the video-only baselines; the paper should state this asymmetry explicitly and report the total frame count per video. The paper also does not report how detection, tracking, and merging influence the final question-answering performance, since Table 6 and Appendix B.3 report only graph quality.
-
The distinction between earlier and updated object states is still not fully explained. How the models use timestamps is not clear. The code retains frame indices, but their use in the final question-answering input remains unclear, particularly when only the predicted graph is provided. This leaves a gap in understanding how the method supports relocation recall and direction questions.
-
The error analysis does not decouple the different factors affecting the results. Appendices D.3 and D.4 provide recall comparisons across models, clustering of frame embeddings by region, and attention analysis, and the doorway-effect analysis in Section 5.3 (Figure 5c) pools T2–T4 without controlling for differences in task composition or path complexity across the room-transition-density quintiles. Prior and updated states also remain tied to the first and second traversals. These observations therefore do not sufficiently establish why cross-room reasoning and state updating remain difficult.
-
Given the training-free design of SPORE, the benchmark would be more informative if the paper could distinguish between “what can be solved through explicit computation and state retrieval without additional training” and “what difficulties remain and might benefit from further training”. Appendix B.2 already scores the predicted MUSHROOM nodes geometrically and Appendix B.4 answers VSI-Bench questions directly from node coordinates. Reporting the corresponding direct-computation accuracy on the MUSHROOM tasks (T2, T3, T7, T8) from the same predicted nodes while keeping the perception outputs fixed would help distinguish coordinate calculation and state retrieval from remaining perception or state-estimation errors, and clarify where additional training might help.
Questions for authors
There are 5 questions about the research details.
-
Please confirm that geometry, detection and tracking run over the full frame sequence while the VLM baselines receive 64 frames, and report the number of frames used at each stage, including the 16 observations per node used for captioning.
-
Could you provide the complete model input for one relocation-recall or relocation-direction question using only the predicted graph, showing how the two candidate locations are presented and how their temporal order can be determined?
-
Appendix B.4 already evaluates answers computed from node coordinates on VSI-Bench. Could you report a similar comparison on MUSHROOM, using the same predicted nodes to compare direct coordinate calculations or time-based state lookup with VLM answers?
-
For relocation questions that require both states, do the sampled frames provide sufficient visual evidence before and after relocation? Please describe how visibility was checked separately for the two traversals.
-
Does accuracy still decrease with room-transition density when comparisons are made within the same task (T2, T3 and T4 separately) and the same scene, given that the density metric is already normalised by the frame gap between the relevant observations?
6 comments
Responses to Reviewer Rhdp (2/4)
W3. Decoupling Factors in the Error Analysis
We are grateful for this detailed suggestion, which helps us further substantiate why cross-region reasoning and state updating remain difficult. We address the two factors raised below.
-
Decoupling Prior and Updated States from Temporal Position: Although prior and updated states are tied to the first and second traversals, Figure 4(b) shows that video-only recall declines with later observation positions even within the same traversal. This indicates that the higher recall of prior states (Appendix D.3) reflects their earlier temporal position rather than a preference for prior states. We will clarify this distinction in the revised manuscript.
-
Doorway Effect within Tasks: To control for task composition, we separately analyze T2, T3, and T4 across the room-transition-density quintiles. Results are reported for Qwen3-VL-8B with video-only inputs, from the 1st quintile (lowest density) to the 5th quintile (highest density). Performance degrades with room-transition density in every task, indicating that the doorway effect consistently appears across tasks (see Q5 for further control of scene-level differences).
Task Q1 Q2 Q3 Q4 Q5 Δ (Q1→Q5) T2: Relative Distance 42.03 32.61 26.09 20.29 16.67 −25.36 T3: Relative Direction 48.62 38.18 27.27 22.02 11.01 −37.61 T4: Static Pathfinding 31.96 32.99 30.93 21.88 10.31 −21.65 Overall 41.28 34.49 27.83 21.28 13.08 −28.20
W4. Explicit Computation vs. Remaining Difficulties
We thank the reviewer for this constructive suggestion, which helps distinguish what can be resolved through explicit computation from what remains for further training.
-
Experimental Setup: Following Appendix B.4, we compute answers to T2, T3, T7, and T8 directly from the same predicted nodes without a VLM, keeping all perception outputs fixed. For T7 and T8, we determine the temporal order of the two retrieved states using the frame indices retained during graph construction. Answer options whose referenced objects are not retrieved are excluded, and questions whose queried object is not retrieved are counted as incorrect.
Method T2: Relative Distance T3: Relative Direction T7: Dynamic Distance T8: Dynamic Direction Direct Computation 68.26 72.39 48.19 45.80 SPORE (Qwen3-VL-8B) 51.88 37.29 47.06 33.28 SPORE (Cambrian-S-7B) 36.96 36.75 35.07 32.82 -
Direction Is Limited by VLM Reasoning: On T3, direct computation reaches 72.39%, whereas Qwen3-VL-8B reaches 37.29% from the same nodes (−35.10%p). This suggests that the VLM struggles to transform world coordinates into the frame of the queried viewpoint, which could be improved through training on reasoning over structured representations.
-
Relocation Is Limited by Retrieving Both States: Direct computation drops by over 20%p on T7 and T8 compared to the static tasks, while Qwen3-VL-8B is only 1.13%p below it on T7. Since a relocated object must be retrieved at both its prior and updated positions, only 41.8% of relocated objects are covered, compared to 74.8% of static objects (Table 7). T8 combines both limitations, making it the most challenging of the four tasks (33.28%).
We believe this distinction strengthens the diagnostic value of MUSHROOM and points to training for reasoning over structured representations as a promising future direction. We will include this analysis in the revised manuscript.
Responses to Reviewer Rhdp (3/4)
Response to Questions:
Q1. Frame Counts at Each Stage
As detailed in W1, SPORE processes the full frame sequence while the VLM baselines receive 64 frames. To disentangle the effect of frame usage, we additionally construct spatial graphs from the same 64 frames given to the VLM, as well as from 128, 256, and 512 frames. Values in parentheses denote gains over video-only inputs with Qwen3-VL-8B (Table 1).
-
Gains Beyond Frame Usage: Even with identical frames, SPORE improves over video-only inputs by +8.05%p, indicating that the gains stem from the structured representation rather than additional frames. Using more frames further improves performance, as denser observations yield more complete graphs.
Frame MRS MRD Overall 64 43.14 (+9.15) 42.95 (+7.30) 43.03 (+8.05) 128 43.94 (+9.95) 43.30 (+7.65) 43.56 (+8.58) 256 45.40 (+11.41) 45.34 (+9.69) 45.37 (+10.39) 512 45.78 (+11.79) 45.28 (+9.63) 45.48 (+10.50)
Q2. Example Input with the Predicted Graph
We thank the reviewer for requesting a concrete example, which helps clarify how relocated objects are presented to the VLM. We provide below the predicted graph input when “white table lamp” is the queried object in dynamic tasks, where it has been relocated in the video.
- Node Retrieval: Following Algorithm 1, SPORE returns up to two nodes for the queried object, where the second lies farther than δ from the first (Appendix B.1). Accordingly, the relocated object is retrieved as two nodes corresponding to its two spatial states, which are provided under the query object in JSON format.
- Excluded Temporal Information: SPORE retains the frame indices and bounding boxes of each node’s observations during graph construction, but excludes them from the input. Node
idis assigned after merging, grouped by category and ordered by the number of detections, and does not reflect temporal order.
"white table lamp": [
{
"id": 155,
"name": "lamp",
"coarse_vote": {"lamp": 1.0},
"center": [0.158, 0.026, 0.101],
"caption": "The object is a table lamp with a dark gray shade, ..., it sits on a light-colored wooden bedside table positioned next to the bed, ...",
"room": "bedroom"
},
{
"id": 151,
"name": "lamp",
"coarse_vote": {"lamp": 1.0},
"center": [0.499, 0.030, -0.122],
"caption": "The masked object is a white table lamp with a pleated, ..., sitting on a white circular dining table in the center of a modern living room. ...",
"room": "living room"
}
]
- Determining Temporal Order: With only the predicted graph, the two candidate states carry no temporal cue, so their order cannot be determined from the graph alone. As described in W2, the VLM determines the order by grounding each state in the video.
Q3. Direct Computation on MUSHROOM
We thank the reviewer for proposing this comparison. Please refer to W4, where we compute answers to T2, T3, T7, and T8 directly from the same predicted nodes, using the retained frame indices for state lookup in T7 and T8.
Responses to Reviewer Rhdp (1/4)
We are grateful to Reviewer Rhdp for the detailed and insightful comments. We are pleased that you found our before-and-after traversal design useful, and that the comparison across video-only inputs, predicted graphs, and ground-truth graphs offers task-specific evidence on where structured representations help.
For weaknesses and questions, we address each point below and will incorporate the clarifications and additional analyses into the revised manuscript.
Response to Weaknesses:
W1. Input Asymmetry and Stage-wise Contribution to QA
We thank the reviewer for pointing out this important point. In the revised manuscript, we will clarify in the experimental setup that SPORE constructs spatial graphs from the full frame sequence, and report frame statistics per video.
Frame Usage across Stages: Each video in MUSHROOM contains 839.3 frames on average (range: 586–1,233). SPORE processes all frames for geometry estimation, detection, and tracking. For captioning, it picks up to 16 observations per node, the most confident one in each time interval, and gives the captioner four of them as full frames with close-up crops of the two largest boxes (Appendix B.1).
Stage-wise Contribution to MUSHROOM (QA Task): To examine how each stage influences the final performance, we provide Qwen3-VL-8B with spatial graphs constructed at each stage of Table 6 and evaluate it on MUSHROOM. Values in parentheses denote gains over video-only inputs with 64 frames. Graphs from each stage progressively improve performance over video-only inputs, with merging particularly benefiting dynamic reasoning, where the MRD gain grows from 4.50%p after tracking to 10.94%p.
We believe the suggested stage-wise analysis offers further insight into how each component of SPORE contributes to long-term spatial reasoning.
W2. Distinguishing Prior and Updated States
We appreciate this question, which allows us to clarify how SPORE supports reasoning about state changes.