acceptodds
Under review as a conference paper at ICLR 2027

Trace-VSI: Benchmarking Progressive Spatial Reasoning Along Egocentric Traversals

Abstract

Understanding a home from egocentric video requires relating new views to previously seen places and accounting for movement between them. Answers to cumulative questions, such as how many distinct rooms have been entered, can therefore change as a traversal continues. End-of-video accuracy alone cannot reveal earlier mistakes or whether predictions track these changes. We introduce TRACE-VSI, a scan-grounded benchmark with 18 full-traversal question types about places, observed routes, and spatial configuration, and repeated questions about 5 cumulative quantities as observations accumulate. At each point, the model is queried independently using video from the start of the traversal up to that point. We score each answer for accuracy and compare predicted changes with changes in the correct answers using Spatial Change Accuracy (SCA). Across 30 systems, 22 exceed the answer-frequency baseline on place/time binding after the full traversal, but only 6 do so on observed-route questions. At earlier and later points, correct count changes often coexist with incorrect counts: Qwen3-VL-8B scores 55.3 SCA on Distinct Rooms but only 13.1% joint exact accuracy. Distance changes are also inaccurate: Qwen2.5-VL-72B has 23.6 m Path Length change MAE, compared with 9.0 m for a video-free prior. Larger frame budgets can improve change agreement without clear evidence of improved joint count correctness. These results reveal persistent count offsets and inaccurate distance increments, showing why accurate changes alone do not establish accurate spatial tracking.

Primary area:
datasets and benchmarks
Keywords:
spatial understanding, video understanding, vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jjsKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study introduces the TRACE-VSI benchmark. Based on 18 types of spatial reasoning questions and 5 types of progressive evaluations, this study reveals that models may correctly identify changes in spatial information. But the models may still fail to accurately estimate actual spatial quantities.

Critical strengths
  1. The benchmark includes a wide range of spatial reasoning tasks, including location recognition, route connectivity, spatial orientation, distance, and area.
  2. This study designs a clear evaluation method, which assesses spatial estimation accuracy at various time points, and also evaluates the changes in these estimates.
Critical weaknesses
  1. For a benchmark study, data accessibility is important. However, Appendix G states only that video metadata is publicly available by default, while the release of the actual videos depends on the licenses of the original data. Also, this study doesn't clearly state whether the QA annotations, ground truth, evaluation code, and video generation pipeline will be made public. It's hard to reproduce or utilize such a benchmark.

  2. TRACE-VSI relies on predefined indoor routes and evaluates spatial reasoning through sampled RGB frames. Although it includes some geometric tasks, these tasks focus mainly on location connectivity and the number of transitions. There is no clear evaluation of directional changes and motion details during these transitions. For example, a 90° left turn and a 90° right turn between locations A and B represent different spatial relationships. But they might show the same connectivity graph. This study does not verify whether the FPS sampling is sufficient to preserve this motion information. Because of this, it's difficult to assess a model's understanding of spatial layouts, motion processes, and the changes.

Questions for authors
  1. If the videos cannot be released, what could be made available? How can the community utilize and continue working on this benchmark?
  2. Could the authors provide additional experimental details? For example, to check whether the model could effectively understand the motion information. If it fails, is it because of the constrained input based on frame sampling? If it does succeed, what specific factors help this understanding? Clarifying these would provide a better understanding of the model's ability.
1 comment

Thank you for the review.

W1/Q1. Data and code availability

Upon publication, we will release:

  • All 950 full-traversal and 1,411 prefix questions, reference answers, and the 953 evaluation pairs.
  • Evaluation code for Tables 2–4, model adapters and prompts, raw predictions from all evaluated systems, and video and observation-window metadata.
  • Question-construction metadata, camera trajectories, and route-generation and rendering code.

Rendered videos will be distributed subject to HM3D’s terms. Users with HM3D access can also regenerate them using the released trajectories and rendering code. The release will therefore support both evaluating new models and reproducing benchmark construction. We will specify these artifacts in Appendix G.

W2/Q2. Direction, motion and frame sampling

TRACE-VSI includes directional information beyond connectivity. Fixed-Axis Bearing asks for the direction between two landmark viewpoints relative to the initial camera heading, with eight answer options. Floor Offset covers vertical displacement. These tasks complement the observed-route graph questions.

To examine sampling sensitivity, we compared the same questions under four input conditions. Mean accuracy (%) over the 26 systems evaluated under all four conditions is:

Question type32 frames64 frames128 frames1 FPS
Fixed-Axis Bearing9.99.69.27.6
Direct Traversal31.129.528.526.6
Destination-Cost Order30.830.931.324.6

Increasing the budget from 32 to 128 frames does not improve mean Bearing accuracy, and only 5/26 systems improve from 32 frames to 1 FPS. Increasing frame density alone therefore did not resolve the directional errors in this comparison. We will add this sampling comparison for all question types.

Reviewer_nwBcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRACE-VSI, a scan-grounded benchmark for evaluating spatial reasoning along egocentric video traversals. It includes 18 question types covering spatial relationships, observed routes, and cumulative spatial quantities. The authors further propose a progressive evaluation protocol and Spatial Change Accuracy (SCA) to assess whether predicted changes agree with ground-truth changes, complementing conventional endpoint accuracy.

Through extensive experiments across 30 multimodal systems, the paper identifies limitations in route reasoning, cumulative spatial estimation, and metric estimation. In particular, the results show that models can predict correct numerical changes while maintaining inaccurate accumulated states. Additional experiments investigate the effects of frame selection and input budgets.

Overall, the paper provides a useful benchmark and informative empirical findings for understanding the spatial reasoning capabilities of current multimodal models.

Critical strengths
  1. A useful perspective on progressive spatial reasoning.**

The paper introduces a meaningful distinction between the correctness of accumulated spatial estimates and the correctness of their changes over time. By combining SCA with endpoint and joint correctness, the benchmark provides a more detailed characterization of model behavior. The finding that models can predict correct changes while maintaining inaccurate accumulated states is particularly interesting and offers a useful perspective beyond conventional endpoint evaluation.

  1. Comprehensive evaluation with informative findings.**

The paper evaluates a diverse set of general-purpose, spatially specialized, and streaming multimodal models across different spatial reasoning tasks. The comparisons with simple priors and the frame-budget experiments provide valuable insights into current model limitations. I also appreciate the additional analyses in the appendix, which help support the interpretation of the results.

Critical weaknesses
  1. The scope and distinctive value of the change-based evaluation could be further clarified.**

Spatial reasoning has been explored in benchmarks such as ScanQA, SQA3D, and OpenEQA, while more recent efforts, including UCS-Bench and Stream3D-Bench, have considered temporally evolving spatial observations. TRACE-VSI extends this research direction through scan-grounded numerical estimation and explicit change-based evaluation. While this is a meaningful contribution, a clearer discussion of the additional diagnostic insights provided by SCA, particularly compared with repeated endpoint evaluation, would further strengthen the paper's positioning.

Additionally, the main SCA evaluation focuses on reference-changing intervals, with all count-based reference increments equal to +1. Although this provides a controlled setting for analyzing prediction changes, spatial tracking also involves recognizing when quantities should remain unchanged, such as during revisits to previously observed rooms. I appreciate that the authors acknowledge this limitation and include stable/changed controls in the appendix. A broader analysis incorporating both types of intervals would help establish the generality of the findings.

  1. Benchmark validation and the robustness of certain experimental conclusions could be strengthened.**

The benchmark relies on automatically generated questions and reference answers derived from scan annotations and trajectory records. While the authors mention manual inspection of a subset of questions in Section 3.2, additional details about the validation procedure, sample size, and annotation quality would help establish confidence in the benchmark.

Some numerical tasks, particularly Path Length and Visited Floor Area, also involve quantities derived from geometric information that is not directly provided to the evaluated models. It would be helpful to clarify whether sufficient metric-scale cues are available in the sampled RGB observations and how the difficulty of these tasks should be interpreted.

Furthermore, certain route-related question types contain relatively few evaluation instances, and Figure 3 shows noticeable answer-position imbalances for some multiple-choice tasks. Although the paper reports relevant baselines, additional discussion of statistical uncertainty and the effect of these imbalances would help strengthen the task-level conclusions.

Questions for authors

Q1. Could the authors further clarify what additional diagnostic insights SCA provides compared with repeated endpoint-based evaluation? In particular, does it reveal distinctive failure patterns or differences in model rankings?

Q2. Could the authors elaborate on whether the main findings remain consistent when reference-stable and reference-changing intervals are evaluated together? The supplementary controls are helpful, and further analysis or discussion would strengthen the interpretation of SCA.

Q3. Could the authors provide more details about the manual validation of automatically generated questions? Additionally, it would be helpful to clarify the availability of metric-scale cues for numerical estimation tasks and whether human performance or another reference baseline could contextualize their difficulty.

Q4. Could the authors comment on the robustness of the route-reasoning results given the relatively small sample sizes of some question types and the answer-position imbalances shown in Figure 3? For example, an analysis relative to the reported majority-position baselines or home-level uncertainty estimates would be informative.

1 comment

Thank you for the review and the concrete suggestions.

Q1. What SCA adds

Endpoint scores can improve without a change in the prediction. On Target Visits, VG-LLM-4B and VG-LLM-8B obtain higher MRA at later prefixes while returning the same answer at both endpoints in every pair (SCA = 0). The reference moves closer to an unchanged prediction.

Similar endpoint scores can also conceal different change accuracy. On Distinct Rooms, InternVL3-14B and Spatial-MLLM have similar early/late MRA (32.5/45.0 and 32.7/42.8, Table 13), but SCA of 53.0 and 13.0.

An additional ranking analysis across the 30 systems gives Kendall’s tau of 0.05 for Target Visits and 0.14 for Path Length between SCA and mean endpoint MRA. Thus, SCA provides information about both model behavior and rankings that endpoint accuracy alone does not convey. Joint correctness further distinguishes correct changes accompanied by correct estimates from correct changes accompanied by incorrect estimates.

Q2. Stable and changing intervals together

We conducted an additional analysis on a separate cohort of 120 shared intervals, containing 176 stable and 140 changing count pairs.

Averaging the three count types equally, the median agreement across 30 systems is 72.6% for preserving stable quantities and 16.6% for reproducing the exact change. When these two agreement rates receive equal weight, only 6/30 systems exceed the 50% score of an always-repeat rule. This balanced agreement score correlates with the main count SCA results (Kendall’s tau = 0.57).

Median joint exact correctness is much lower: 17.8% on stable pairs and 3.8% on changing pairs. The distinction between consistent responses and correct accumulated estimates therefore persists in this control, including when the reference remains unchanged.

Q3. Validation, metric-scale cues and baselines

Manual validation consisted of author inspection of sampled questions through visualizations, alongside automated checks of source provenance and record consistency. We will describe the author inspection explicitly in Sec. 3.2.

The metric tasks ask models to estimate quantities from RGB observations, where familiar object sizes and motion across views provide cues for scale estimation. Scan geometry and recorded trajectories determine the reference distances and areas; models receive the RGB observations.

Table 4 provides reference baselines for interpreting the results: the video-free prior has change MAE of 9.0 m for Path Length and 4.6 m² for Visited Floor Area. For comparison, Qwen2.5-VL-72B has Path Length change MAE of 23.6 m.

Q4. Route robustness

We performed an additional home-cluster bootstrap analysis with 4,000 resamples. After subtracting each group’s answer-frequency baseline, the mean advantage for place/time binding over route questions is +19.8 percentage points (95% CI [+9.5, +27.7]). No system’s margin over the route-group majority-position baseline has a confidence interval entirely above zero.

Excluding Shortest-Route First Hop leaves 11/30 systems above the route frequency baseline, compared with 22/30 for place/time binding. The group-level contrast therefore persists after removing the smallest and most imbalanced type. We will include these results and the individual types’ uncertainty intervals in the revision.

Reviewer_AptAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current video benchmarks only score spatial questions at the very end of a traversal, hiding intermediate mistakes and failing to test whether an AI actually updates its mental map as it moves. The authors introduce TRACE-VSI, a benchmark that repeatedly asks models cumulative questions at multiple points along a walk to evaluate whether their predictions track step-by-step changes rather than just getting lucky at the end.

Critical strengths

The paper makes good contribution by decoupling change accuracy (Spatial Change Accuracy, SCA) from joint endpoint correctness. Showing that high increment accuracy frequently coexists with persistent count offsets is a valuable observation that exposes the vulnerability of relying purely on step-level transitions.

The inclusion of a video-free empirical prior provides a necessary reality check for multimodal foundation models. Showing that state-of-the-art models (such as Qwen2.5-VL-72B with 23.6 m MAE on Path Length) perform substantially worse than a naive dataset prior highlights severe deficiencies in physical metric grounding that standard vision-language benchmarks overlook.

Critical weaknesses

1- The main concern is novelty. The paper does not propose a new model or learning method; it only introduces an evaluation benchmark that tests existing models by asking questions at multiple points along a video. Specifically, they only test existing models like Qwen2.5-VL, Qwen3-VL, InternVL3, LLaVA, Cambrian, SenseNova, VG-LLM, Dispider, and OASIS. Furthermore, 3D spatial benchmarking is already well-explored by prior papers such as OpenEQA, ScanQA, SQA3D, and Stream3D-Bench.

2-The benchmark's execution does not substantiate its claims of evaluating "progressive spatial reasoning" or "streaming spatial intelligence." The benchmark simply evaluates video clips from scratch at each step, rather than letting a model pass an updated memory state over time.

3-The paper concludes the models' inability to perform cognitive mapping and observed-route graph reasoning. However, as documented in Figure 2(a), several primary topological categories contain critically small question sets. Making conclusions by using just 30 frontier model families from 12 to 22 multiple-choice samples lacks statistical power and is highly vulnerable to random choice variance.

4-The main conceptual finding is that a model can get the +1 change right while its starting count is already wrong. I would assume that this is simply a standard systematic bias or constant offset, not a new concept in machine learning. Aside from rendering paths in existing 3D scans and generating question templates, the paper does not propose any new model, architecture, or training method.

Questions for authors

Please see above for the main point. Also, some question types have only 12 or 22 questions in total (like Shortest-Route First Hop and Mandatory Place). How can we find strong conclusions about route reasoning from such a small sample?

1 comment

Thank you for the review.

W1/W4. Contribution and count offsets

TRACE-VSI is a benchmark and evaluation protocol. Compared with OpenEQA, ScanQA, SQA3D and Stream3D-Bench, it asks the same cumulative question at two points of a traversal, recomputes the reference at each, and scores the change together with both answers.

We checked whether a constant offset explains a correct +1 with wrong counts. If each system had a constant offset, shifting all of its answers by one value would make both answers exact whenever the +1 is correct. Even when we fit this shift on the evaluation pairs themselves, median joint correctness across the 30 systems rises only from 3.2% to 11.5%, while median SCA is 30.6. This suggests that the offsets vary across questions.

W2. Progressive evaluation and memory

Independent-prefix evaluation assesses the predictions a system produces as observations accumulate. It does not, by itself, establish that the system maintains and updates a persistent internal map.

The paired evaluation remains informative under this protocol: whether a system recomputes an estimate or updates a stored representation, its predicted change can be compared with the reference change. The finding that correct increments coexist with incorrect counts concerns this observable relationship between outputs.

We will distinguish this evaluation of progressive spatial estimates from evaluation of persistent streaming memory, and revise the broad “streaming spatial intelligence” wording accordingly.

W3/Question. Route sample sizes

Our group-level comparison uses 126 observed-route questions and 214 place/time-binding questions. To address the sample-size concern, we performed an additional home-cluster bootstrap analysis with 4,000 resamples, keeping questions from the same home together.

After subtracting each group’s answer-frequency baseline, all 30 systems have a larger margin on place/time binding. The mean difference is +19.8 percentage points, with a 95% CI of [+9.5, +27.7].

The comparison also persists after excluding Shortest-Route First Hop, the smallest type: 11/30 systems exceed the remaining route group’s frequency baseline, versus 22/30 for place/time binding. We will report the sensitivity results and individual types’ uncertainty intervals in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.