acceptodds
Under review as a conference paper at ICLR 2027

TrajsCogMap-Bench: Evaluating Cognitive Mapping and Spatial Memory from Egocentric Video Trajectories

Abstract

Navigation requires more than remembering visited places: agents must integrate egocentric observations into spatial knowledge that supports subsequent queries and route planning. Evaluating this ability requires distinguishing local spatial understanding, integration across trajectories, and the retention of spatial information for future use. We introduce \textbf TrajsCogMap-Bench, an open-loop benchmark for evaluating cognitive mapping and spatial memory in multimodal large language models. Built from real-world egocentric videos in indoor environments rich in points of interest, the benchmark organizes spatial queries and route-prediction tasks across three levels: local motion understanding, single-trajectory integration, and multi-trajectory reasoning. The multi-trajectory setting probes spatial knowledge construction and use from multiple traversals of the same environment, covering different locations and viewpoints, including route inference beyond directly experienced paths. We report answer accuracy for all tasks except open-ended route navigation. For the latter, an executor model follows the generated instructions in a simulator grounded in the environment topology and egocentric observations, with performance measured by success rate (SR) and success weighted by path length (SPL). We provide complementary full-context and online-memory protocols: full-context evaluation tests joint integration of observations available at answering time, while online-memory evaluation tests the use of memory built incrementally from sequential observations without knowledge of future questions. Our evaluations reveal a gap between remembering landmarks and reasoning about their spatial relations to plan routes. Performance further declines when models rely on incrementally constructed memory, highlighting limitations in how current memory systems preserve spatial information for subsequent reasoning.

Primary area:
datasets and benchmarks
Keywords:
Cognitive Mapping, Spatial Memory, Egocentric Video Understanding, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SdnhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TrajsCogMap-Bench, an open-loop benchmark for cognitive mapping and spatial memory from real-world egocentric observations. Its contributions include an annotated dataset of 1,921 questions, a three-level task hierarchy spanning local motion through multi-trajectory reasoning, route inference through visible but untraversed connections, and paired full-context and question-independent online-memory protocols.

Validation consists of evaluating seven MLLMs using answer accuracy and executor-based SR/SPL for open-ended routes. Text and image–text memory are compared with full-context access using Qwen3.8-Max, excluding local-motion tasks. The authors report stronger landmark recall than spatial-relation and route performance, with further declines under the evaluated memory configurations.

Critical strengths

A well-motivated evaluation axis. The benchmark groups multiple trajectories of the same real-world indoor environment and includes route inference through connections that are visible in the observations but never traversed. This targets a core property of cognitive maps: integrating spatial evidence across experiences to infer routes beyond directly experienced paths. The annotated topology also supports programmatically derived reference answers.

Critical weaknesses

W1. The distinctive evaluation setting is not sufficiently validated, and important uncertainties remain in the reported evaluation.

  • The novel setting is not evaluated separately. Results are aggregated by task type, without a breakdown by single- vs. multi-trajectory input or by questions requiring untraversed connections. The paper provides no dedicated evidence-dependency analysis or trajectory-removal ablation validating that the relevant questions genuinely require information from multiple trajectories.

  • Several results are below chance without explanation. On the 4-way POI Direction and Frame Direction tasks, Qwen3-VL scores 5.2–11.8% and 14.0–18.1%, respectively, against a 25% chance level. These unusually low scores warrant an audit of answer parsing, option distributions, direction conventions, and labels. The paper does not establish whether they reflect systematic model failures or evaluation artifacts.

  • Parts of the manuscript remain unfinished. Table 2 marks the overall scores for Qwen3.8-Max and Kimi K3 as provisional estimates pending verification of exact counts. Table 3 likewise reports provisional overall scores requiring verification of evaluation denominators and correct-answer counts. The appendix examples in Figure 6(b) also contain unfinished placeholders.

  • Baselines are missing. There are no chance, human, or blind (text-only) baselines, although many comparable benchmarks report such baselines. These would help establish task difficulty and reveal whether questions can be answered without the intended visual evidence.

  • The evaluation depends heavily on one model. Qwen3.8-Max is both an evaluated instruction generator and the sole route executor, leaving open-route rankings potentially dependent on executor-specific behavior. Reference instructions achieve only 70.3% SR, confirming that execution failures contribute to the scores even with reference guidance. The memory evaluation also uses only this backbone, limiting the generality of its findings.

W2. Substantial overlap with prior work and limited diagnostic value of the memory comparison.

  • Memory protocol. Building memory without knowing future questions is already the native setting of ReMEmbR. ReMEmbR and RAVEN also compare memory-based answering against full-context access. The question-independent memory protocol itself therefore does not establish a distinct contribution.

  • Tasks and findings. OST-Bench already includes displacement, rotation, and visible-information tasks, and reports substantially stronger visible-information performance than agent-state and agent–object spatial reasoning. It also reports declining accuracy as online observation histories grow, although this is not the same comparison as external memory versus full-context access here. The paper does not cite OST-Bench or clearly distinguish its overlapping tasks and findings from that work.

  • Memory baselines. The evaluated configurations differ substantially from the original systems: they omit explicit position- and timestamp-based retrieval, while the image–text variant retains a caption-and-summary hierarchy that RAVEN avoids. These differences are not necessarily inappropriate for the benchmark's input constraints, but the configurations should not be interpreted as faithful evaluations of ReMEmbR or RAVEN. Without ablations of representation and retrieval, the experiments do not identify why memory reduces performance or why the two variants achieve similar overall scores (60.3 vs. 59.5).

  • Bibliography errors. Some bibliography entries have incorrect author information. For example, VSI-Bench's first author is Jihan Yang, not Ruochen Zhang, and REM's first author is Jacob Thompson, not Smith. The CapNav entry's arXiv identifier (2511.06881) points to an unrelated paper. These errors undermine confidence in the positioning against prior work and warrant a comprehensive bibliography check.

Questions for authors

Q1. Could you report results separately for single-trajectory questions, multi-trajectory questions, and route questions involving visible but untraversed connections, using accuracy and SR/SPL where applicable? How many questions genuinely require cross-trajectory evidence or an untraversed connection, and how is this dependency verified?

Q2. On the 4-way POI Direction and Frame Direction tasks, the Qwen3-VL models achieve 5.2–11.8% and 14.0–18.1%, respectively, below the 25% chance level. Could you investigate whether these scores reflect systematic model failures, answer-parsing issues, option distributions, or inconsistencies in direction conventions and labels? Could you also report the correct-option and predicted-option distributions, per-task chance baselines where applicable, and blind (text-only) baselines?

Q3. Table 2 marks the overall scores for Qwen3.8-Max and Kimi K3 as provisional estimates, and Table 3 also reports provisional overall scores. Could you provide verified pooled accuracies, evaluation denominators, and exact correct-answer counts? Could you additionally report human performance on a representative subset to help establish task difficulty?

Q4. Qwen3.8-Max is both an evaluated instruction generator and the shared route executor. Given that reference instructions achieve only 70.3% SR, could you provide an execution-failure analysis or a small-scale comparison with another executor to assess whether executor-specific behavior affects model rankings? Do rankings remain stable if generated instructions are normalized into a common format while preserving their meaning?

Q5. The memory configurations omit explicit position- and timestamp-based retrieval, while the image–text variant retains a caption-and-summary hierarchy unlike RAVEN. Could you explain these adaptations under the benchmark’s input constraints and provide evidence about how representation and retrieval choices affect the full-context versus memory gap? What explains the similar overall scores of the two variants (60.3 vs. 59.5), and how robust are the findings to an additional backbone?

Q6. OST-Bench [1] includes displacement/rotation tasks and reports weak agent-state reasoning and declining accuracy as online observation histories grow. Cambrian-S [2] studies long-horizon spatial recall and continual streaming evaluation. These closely related works are not discussed. Could you clarify what additional capabilities and empirical findings this benchmark establishes beyond them, while distinguishing their protocols from your question-independent external-memory setting?

[1] Lin, Jingli, et al. "Ost-bench: Evaluating the capabilities of mllms in online spatio-temporal scene understanding." Advances in Neural Information Processing Systems 38 (2026). [2] Yang, Shusheng, et al. "Cambrian-s: Towards spatial supersensing in video." International Conference on Learning Representations. Vol. 2026. 2026.

Reviewer_aTpZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TrajsCogMap-Bench, a benchmark for evaluating cognitive mapping and spatial memory in multimodal large language models using real-world egocentric video trajectories. The benchmark covers three levels of spatial reasoning: local motion understanding, single-trajectory integration, and multi-trajectory reasoning. It also introduces full-context and online-memory evaluation protocols to assess spatial reasoning with direct access to historical observations or incrementally constructed memory. The experiments reveal limitations in spatial reasoning, route planning, and the use of spatial memory in existing models.

Critical strengths
  1. This paper extends existing spatial reasoning benchmarks by introducing tasks that evaluate cross-trajectory spatial integration and route inference beyond directly traversed paths. These tasks expand the scope of existing evaluations beyond landmark recognition and recall.

  2. This paper constructs the benchmark using real-world indoor observations and provides topological, geometric, and landmark annotations. These annotations support objective and verifiable evaluation of spatial reasoning.

Critical weaknesses
  1. This paper does not sufficiently validate whether the proposed benchmark measures the intended cognitive mapping capabilities. Correct answers to spatial questions and route inference tasks do not establish that models have constructed globally consistent cognitive maps. Moreover, the paper does not demonstrate whether improvements on the benchmark translate into better downstream navigation performance. These limitations raise fundamental concerns about the validity and practical significance of the proposed benchmark.

  2. This paper does not adequately evaluate the role of cognitive mapping and spatial memory in embodied navigation. The fixed-trajectory, open-loop setting excludes active exploration, online decision-making, and error recovery. Furthermore, the navigation evaluation relies on a separate executor, making navigation success dependent on both route planning quality and executor performance. Even with ground-truth instructions, the executor achieves only a 70.3% success rate. These limitations prevent the reported results from directly reflecting the evaluated models' navigation capabilities.

Questions for authors
  1. What evidence demonstrates that the proposed benchmark measures cognitive mapping capabilities rather than general spatial question-answering performance, and that these capabilities are predictive of downstream navigation performance?

  2. Can the authors provide evidence that the multi-trajectory tasks genuinely require integrating information across different trajectories, rather than relying on information from a single trajectory?

  3. Can the authors evaluate route planning quality independently of the navigation executor to distinguish planning failures from execution failures?

Reviewer_LtgJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces TrajsCogMap-Bench, a benchmark for evaluating cognitive mapping and spatial memory in multimodal language models using egocentric video trajectories. The benchmark consists of 1,921 questions across nine tasks, covering local motion understanding, landmark recall, spatial relationships, and route planning across multiple trajectories. The authors evaluate seven MLLMs under a full-context setting and further compare this with two online-memory configurations. The results suggest that current models struggle with spatial reasoning and route planning, particularly when relying on incrementally constructed memory.

Critical strengths

None, the paper is AI-generated, with hallucinated numbers.

Critical weaknesses
  1. "Qwen3.8-Max and Kimi K3 overall values remain provisional weighted estimates pending verification of exact counts."

  2. "Overall values are provisional weighted estimates from the reported task accuracies and task counts; final pooled scores require verification of evaluation denominators and exact correct-answer counts."

  3. "[Insert the verified reference instruction]"

The paper is clearly not ready. The authors didn't even read the paper once!

The scores are not mathematically correct for example: 86.40% visited-POI recall accuracy over 83 questions. This cannot result from an integer number of correct answers. The nearest possibilities are 85.54% and 86.75%

Questions for authors

N.A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.