- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study introduces the TRACE-VSI benchmark. Based on 18 types of spatial reasoning questions and 5 types of progressive evaluations, this study reveals that models may correctly identify changes in spatial information. But the models may still fail to accurately estimate actual spatial quantities.
Critical strengths
- The benchmark includes a wide range of spatial reasoning tasks, including location recognition, route connectivity, spatial orientation, distance, and area.
- This study designs a clear evaluation method, which assesses spatial estimation accuracy at various time points, and also evaluates the changes in these estimates.
Critical weaknesses
-
For a benchmark study, data accessibility is important. However, Appendix G states only that video metadata is publicly available by default, while the release of the actual videos depends on the licenses of the original data. Also, this study doesn't clearly state whether the QA annotations, ground truth, evaluation code, and video generation pipeline will be made public. It's hard to reproduce or utilize such a benchmark.
-
TRACE-VSI relies on predefined indoor routes and evaluates spatial reasoning through sampled RGB frames. Although it includes some geometric tasks, these tasks focus mainly on location connectivity and the number of transitions. There is no clear evaluation of directional changes and motion details during these transitions. For example, a 90° left turn and a 90° right turn between locations A and B represent different spatial relationships. But they might show the same connectivity graph. This study does not verify whether the FPS sampling is sufficient to preserve this motion information. Because of this, it's difficult to assess a model's understanding of spatial layouts, motion processes, and the changes.
Questions for authors
- If the videos cannot be released, what could be made available? How can the community utilize and continue working on this benchmark?
- Could the authors provide additional experimental details? For example, to check whether the model could effectively understand the motion information. If it fails, is it because of the constrained input based on frame sampling? If it does succeed, what specific factors help this understanding? Clarifying these would provide a better understanding of the model's ability.
Thank you for the review.
W1/Q1. Data and code availability
Upon publication, we will release:
Rendered videos will be distributed subject to HM3D’s terms. Users with HM3D access can also regenerate them using the released trajectories and rendering code. The release will therefore support both evaluating new models and reproducing benchmark construction. We will specify these artifacts in Appendix G.
W2/Q2. Direction, motion and frame sampling
TRACE-VSI includes directional information beyond connectivity. Fixed-Axis Bearing asks for the direction between two landmark viewpoints relative to the initial camera heading, with eight answer options. Floor Offset covers vertical displacement. These tasks complement the observed-route graph questions.
To examine sampling sensitivity, we compared the same questions under four input conditions. Mean accuracy (%) over the 26 systems evaluated under all four conditions is:
Increasing the budget from 32 to 128 frames does not improve mean Bearing accuracy, and only 5/26 systems improve from 32 frames to 1 FPS. Increasing frame density alone therefore did not resolve the directional errors in this comparison. We will add this sampling comparison for all question types.