- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TrajsCogMap-Bench, an open-loop benchmark for cognitive mapping and spatial memory from real-world egocentric observations. Its contributions include an annotated dataset of 1,921 questions, a three-level task hierarchy spanning local motion through multi-trajectory reasoning, route inference through visible but untraversed connections, and paired full-context and question-independent online-memory protocols.
Validation consists of evaluating seven MLLMs using answer accuracy and executor-based SR/SPL for open-ended routes. Text and image–text memory are compared with full-context access using Qwen3.8-Max, excluding local-motion tasks. The authors report stronger landmark recall than spatial-relation and route performance, with further declines under the evaluated memory configurations.
Critical strengths
A well-motivated evaluation axis. The benchmark groups multiple trajectories of the same real-world indoor environment and includes route inference through connections that are visible in the observations but never traversed. This targets a core property of cognitive maps: integrating spatial evidence across experiences to infer routes beyond directly experienced paths. The annotated topology also supports programmatically derived reference answers.
Critical weaknesses
W1. The distinctive evaluation setting is not sufficiently validated, and important uncertainties remain in the reported evaluation.
-
The novel setting is not evaluated separately. Results are aggregated by task type, without a breakdown by single- vs. multi-trajectory input or by questions requiring untraversed connections. The paper provides no dedicated evidence-dependency analysis or trajectory-removal ablation validating that the relevant questions genuinely require information from multiple trajectories.
-
Several results are below chance without explanation. On the 4-way POI Direction and Frame Direction tasks, Qwen3-VL scores 5.2–11.8% and 14.0–18.1%, respectively, against a 25% chance level. These unusually low scores warrant an audit of answer parsing, option distributions, direction conventions, and labels. The paper does not establish whether they reflect systematic model failures or evaluation artifacts.
-
Parts of the manuscript remain unfinished. Table 2 marks the overall scores for Qwen3.8-Max and Kimi K3 as provisional estimates pending verification of exact counts. Table 3 likewise reports provisional overall scores requiring verification of evaluation denominators and correct-answer counts. The appendix examples in Figure 6(b) also contain unfinished placeholders.
-
Baselines are missing. There are no chance, human, or blind (text-only) baselines, although many comparable benchmarks report such baselines. These would help establish task difficulty and reveal whether questions can be answered without the intended visual evidence.
-
The evaluation depends heavily on one model. Qwen3.8-Max is both an evaluated instruction generator and the sole route executor, leaving open-route rankings potentially dependent on executor-specific behavior. Reference instructions achieve only 70.3% SR, confirming that execution failures contribute to the scores even with reference guidance. The memory evaluation also uses only this backbone, limiting the generality of its findings.
W2. Substantial overlap with prior work and limited diagnostic value of the memory comparison.
-
Memory protocol. Building memory without knowing future questions is already the native setting of ReMEmbR. ReMEmbR and RAVEN also compare memory-based answering against full-context access. The question-independent memory protocol itself therefore does not establish a distinct contribution.
-
Tasks and findings. OST-Bench already includes displacement, rotation, and visible-information tasks, and reports substantially stronger visible-information performance than agent-state and agent–object spatial reasoning. It also reports declining accuracy as online observation histories grow, although this is not the same comparison as external memory versus full-context access here. The paper does not cite OST-Bench or clearly distinguish its overlapping tasks and findings from that work.
-
Memory baselines. The evaluated configurations differ substantially from the original systems: they omit explicit position- and timestamp-based retrieval, while the image–text variant retains a caption-and-summary hierarchy that RAVEN avoids. These differences are not necessarily inappropriate for the benchmark's input constraints, but the configurations should not be interpreted as faithful evaluations of ReMEmbR or RAVEN. Without ablations of representation and retrieval, the experiments do not identify why memory reduces performance or why the two variants achieve similar overall scores (60.3 vs. 59.5).
-
Bibliography errors. Some bibliography entries have incorrect author information. For example, VSI-Bench's first author is Jihan Yang, not Ruochen Zhang, and REM's first author is Jacob Thompson, not Smith. The CapNav entry's arXiv identifier (2511.06881) points to an unrelated paper. These errors undermine confidence in the positioning against prior work and warrant a comprehensive bibliography check.
Questions for authors
Q1. Could you report results separately for single-trajectory questions, multi-trajectory questions, and route questions involving visible but untraversed connections, using accuracy and SR/SPL where applicable? How many questions genuinely require cross-trajectory evidence or an untraversed connection, and how is this dependency verified?
Q2. On the 4-way POI Direction and Frame Direction tasks, the Qwen3-VL models achieve 5.2–11.8% and 14.0–18.1%, respectively, below the 25% chance level. Could you investigate whether these scores reflect systematic model failures, answer-parsing issues, option distributions, or inconsistencies in direction conventions and labels? Could you also report the correct-option and predicted-option distributions, per-task chance baselines where applicable, and blind (text-only) baselines?
Q3. Table 2 marks the overall scores for Qwen3.8-Max and Kimi K3 as provisional estimates, and Table 3 also reports provisional overall scores. Could you provide verified pooled accuracies, evaluation denominators, and exact correct-answer counts? Could you additionally report human performance on a representative subset to help establish task difficulty?
Q4. Qwen3.8-Max is both an evaluated instruction generator and the shared route executor. Given that reference instructions achieve only 70.3% SR, could you provide an execution-failure analysis or a small-scale comparison with another executor to assess whether executor-specific behavior affects model rankings? Do rankings remain stable if generated instructions are normalized into a common format while preserving their meaning?
Q5. The memory configurations omit explicit position- and timestamp-based retrieval, while the image–text variant retains a caption-and-summary hierarchy unlike RAVEN. Could you explain these adaptations under the benchmark’s input constraints and provide evidence about how representation and retrieval choices affect the full-context versus memory gap? What explains the similar overall scores of the two variants (60.3 vs. 59.5), and how robust are the findings to an additional backbone?
Q6. OST-Bench [1] includes displacement/rotation tasks and reports weak agent-state reasoning and declining accuracy as online observation histories grow. Cambrian-S [2] studies long-horizon spatial recall and continual streaming evaluation. These closely related works are not discussed. Could you clarify what additional capabilities and empirical findings this benchmark establishes beyond them, while distinguishing their protocols from your question-independent external-memory setting?
[1] Lin, Jingli, et al. "Ost-bench: Evaluating the capabilities of mllms in online spatio-temporal scene understanding." Advances in Neural Information Processing Systems 38 (2026). [2] Yang, Shusheng, et al. "Cambrian-s: Towards spatial supersensing in video." International Conference on Learning Representations. Vol. 2026. 2026.