- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors carefully designed a series of experiments to evaluate the spatial cognition capabilities of large language models (LLMs). Drawing on behavioural psychology and computational simulation, the study examined planning, working memory, and egocentric navigation mechanisms. The findings suggest that leading models substantially outperform mid-tier and smaller models. Their behaviour extends beyond random or naive moves and is consistent with the use of internalised, algorithm-like heuristics.
Critical strengths
The authors designed the experiments very carefully. In particular, the framework works on closed models with no need of weight access, internal activations. By fitting against execution traces, the synthesised code explains why models fail rather than assuming an optimal solver. Uses situational forward Kullback-Leibler divergence to normalise for model entropy and cross-validates continuous/discrete parameter bounds via coordinate descent.
Critical weaknesses
In 3D egocentric navigation, programs do not process raw images directly; instead, camera views an approximated top-down 2D occupancy grid, creating a representation gap between VLM visual inputs and program inputs.
Synthesized programs describe what behavioral rules reproduce the model's choices, but do not explain how these heuristics are physically represented within internal transformer attention heads or circuits.
Data and code are not available.
Questions for authors
Why use LLMs whose source codes are not avaible, such as GPT 5.4, Anonymous-6rhc4, Gemma 4 31B, Qwen 3.6 27B, Qwen 3.5 4B, and InternVL 3.5 38B? Using opensource LLMs will be better to guarantee the conditions of the experiments.
Are data and code available? How can we reproduce the results?
2 comments
Feedback
Hi, the review deadline is approaching. Please take a look at the authors' rebuttal and post your feedback (and update your rating if appropriate) at your earliest convenience. Thank you! AC
Official author response to Reviewer qQ7P (1/3): Representation gap in 3D egocentric navigation
1. Representation gap in 3D egocentric navigation
Egocentric navigation is particularly important to analyze now because recent frontier VLMs exhibit a sharp rise in spatial competence (Section 1). These tasks test multi-turn decision-making, perceptual grounding, and visual navigation (Section 2)—central emerging capabilities whose behavioral strategies are worth characterizing. At the same time, we agree that the representation gap between raw egocentric RGB images and top-down 2D grids is a genuine challenge.
To narrow this gap, we use the 3D scene and camera poses to reconstruct visibility-masked local occupancy maps with explicit occlusion, landmark metadata, and fitted perceptual limits such as field-of-view, depth, and waypoint subsampling constraints (Appendices B.1 and B.2). Because this reduces rather than eliminates perceptual mismatch, we scope our claims to geometry- and action-level behavior rather than raw vision. Even under this approximation, evolved programs fit egocentric behavior far better than the best seed hypothesis: on Route Retracing and Shortcut Discovery they raise next-action accuracy from 41.9% to 56.2% and from 42.2% to 55.8%, and lower situational KL from 0.31 to 0.05 and 0.07 nats (Section 4.2 and Appendix A.3). The gains hold on unseen 3D scenes, with test accuracy of 49.7% vs. 38.2% and 59.2% vs. 44.5% (Appendix A.2).