- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SpatialAgent, which maintains an explicit incremental map for LLM agents operating in partially observable environments. Experiments span four environments, multiple language models, and several map-construction strategies. Although the method sometimes produces improvements, its contribution is mainly a system-level integration of established spatial-map ideas. The current results do not support the paper’s broad claims about novelty, generalization, working memory, or consistent superiority. Overall, this paper does not meet the acceptance criteria of ICLR.
Critical strengths
- The motivation is reasonable. Maintaining structured spatial state is highly relevant to long-horizon agents under partial observability, and the explicit map provides an interpretable mechanism for storing spatial information.
- The paper evaluates several model scales across four environments and compares map-construction strategies. Some configurations show large improvements, such as Qwen3-4B on BabyAI, indicating that explicit spatial state can be valuable under suitable conditions.
Critical weaknesses
- Unclear novelty and inaccurate conceptual positioning. Explicit spatial maps have already been used to improve LLM agents, while SpatialAgent largely adopts a conventional location–relation–property representation. Moreover, only the map data structure is shared across environments, each environment still requires specialized procedures. Therefore, the claimed generalizability is not convincingly demonstrated.
- The empirical evidence does not support the strength of the claims. SpatialAgent is not consistently superior. For example, it underperforms HiAgent on BabyAI with Qwen3-8B and Qwen3-32B, ReAct on TextWorld with Qwen3-8B, and StateAct on BabaIsAI with Qwen3-32B. More importantly, the evaluation uses the default BALROG episode counts with only a single attempt per episode, and the paper does not report independent repeated runs across random seeds. Although Appendix Table 4 reports standard errors for the baseline methods, corresponding uncertainty estimates are not provided for SpatialAgent, the map-construction comparison, or the main ablations. Consequently, the claim that SpatialAgent “significantly outperforms” prior methods is not statistically substantiated by paired tests, confidence intervals for performance differences, or other appropriate analyses. The absence of a clear limitations section further obscures the method’s environment-specific engineering requirements and inconsistent performance.
Questions for authors
- Compared with explicit-map or graph-memory approaches such as AriGraph and MapGPT, what is the essential and non-substitutable contribution of SpatialAgent?
- If each environment still requires a specialized representation schema and MapBuilder, what exactly do the authors mean by “generalization”?
- Could the authors report the accuracy of the constructed maps, including nodes, edges, object attributes, and agent locations. Why direct LLM-based map construction yields a score of 0.0 in some settings?
- When output tokens, total interaction steps, and all map-construction calls are included, does SpatialAgent still achieve a lower total inference cost?