acceptodds
Withdrawn from ICLR 2027

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack post-hoc re-observability. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose GSMem, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with Spatial Recollection: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to “hallucinate” optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied exploration, Vision Language Navigation, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_57QAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GSMem, a zero‑shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). It aims to address the problem that conventional spatial memory struggles to retrieve visual evidence of historical scenes. Existing scene‑graph or keyframe‑based memory generally relies on object detection outputs or fixed viewpoints obtained during initial observations. When a target is misdetected or only observed from unfavorable viewpoints, the agent may fail to recover relevant information from its stored memory.

Critical strengths

This paper proposes GSMem, a zero-shot embodied exploration and reasoning framework that uses 3D Gaussian Splatting (3DGS) as persistent spatial memory. It enables agents to re-observe previously explored regions from novel viewpoints without physically revisiting them. The framework combines object-level and semantic-level retrieval, optimal viewpoint selection, and a hybrid exploration strategy based on semantic relevance and geometric information gain. Experiments on OpenEQA and GOAT-Bench show consistent improvements over existing baselines, supported by ablation studies and additional evaluations of robustness and efficiency.

Critical weaknesses
  1. The paper presents 3DGS-based persistent spatial memory and post-hoc re-observation as key contributions. However, similar ideas have already been explored in GaussNav [1], 3DGSNav [2], and GLMap [3]. In particular, 3DGSNav also uses 3DGS as spatial memory and renders virtual views to assist navigation. Therefore, the use of 3DGS for scene re-observation alone does not establish sufficient novelty.
  2. The geometric information-gain score uses the trace of the incremental Fisher Information Matrix, which does not explicitly account for information already captured in the current map. It is therefore unclear whether this score accurately reflects the information gained from new observations. Moreover, the paper does not explain how the incremental Fisher matrix is estimated for unexplored frontiers. Although the ablation results show that hybrid exploration improves performance, they do not isolate the contribution of the proposed information-gain metric.

[1] Lei, Xiaohan, et al. "Gaussnav: Gaussian splatting for visual navigation." IEEE Transactions on Pattern Analysis and Machine Intelligence 47.5 (2025): 4108-4121.

[2] Zheng, Wancai, et al. "3DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting." arXiv preprint arXiv:2602.12159 (2026).

[3] Zhang, Sixian, et al. "Multi-Scale Gaussian-Language Map for Zero-shot Embodied Navigation and Reasoning." arXiv preprint arXiv:2605.01736 (2026).

Questions for authors

Since 3DGSNav already leverages persistent 3DGS memory, free‑view rendering, and target re‑verification under virtual views to assist navigation decision‑making, authors are requested to clarify the essential differences in core mechanisms between GSMem and 3DGSNav, elaborate on the capabilities enabled by the proposed multi‑level retrieval and viewpoint‑selection mechanism that cannot be achieved by existing systems, and add detailed methodological comparisons against GaussNav, 3DGSNav and GLMap, with direct performance comparisons under identical experimental settings whenever feasible.

Reviewer_bq9AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an embodied exploration and reasoning task using 3D Gaussian Splatting. Especially the authors focus on the "post-hoc re-observability" (I do not clearly understand this terminology) and this problem is indirectly described as:

  • Line 51 - Line 53 : "We identify a fundamental gap underlying these limitations: the lack of post-hoc re-observability. Unlike humans, who can mentally re-visit a past scene from a new perspective to discover missed details, current agents are "locked" into the specific observations they made during their initial exploration process.

It is highly confusing whether the proposed method requires a pre-processing step to build memory, but I guess the answer may be "Yes".

Proprocessing

Given streaming RGB-D, the proposed method reconstruct 3D scenes in 3D Gaussian representations where each 3D Gaussian also involve a high-dimensional semantic feature vector distilled from CLIP (Line 183). Also, the authors build a scene graph during this process (Figure 2).

Inference

Given an instruction, the proposed pipeline leverages off-the-shelf VLMs to sample the viewpoint candidates.

That is my understanding so far. But I am not sure about section 3.4 Hybrid Exploration Strategy. The paper is not well written. It is highly confusing me a lot.

Critical strengths

Honestly, I failed to find any strengths of this paper. The authors addressed the importance of 3DGS as 3D memory for the target task, but I do not feel that is quite novel enough. The framework is working under the simulation and the VLMs can take rollouts and judge which new viewpoint will be much informative or not. So, introducing the novel-view synthesis into this task is quite heuristic in my opinion.

Critical weaknesses

Why used 3DGS?

The proposed pipeline is highly similar to 3D-Mem and SplatReasoner. Moreover, I do not understand why 3D Gaussian Splatting is used in this pipeline. Do authors have specific reasons to use this representation? For example in SplatReasoner, they use 3DGS to adjust the viewpoint that describe the best visual information from the given instructions. To do the novel-view synthesis, 3DGS is needed. But for this paper, I have no idea. Even PointMap from VGGT or Depth-Anything-v3 can be applicable into this pipeline.

What is the true meaning of "post-hoc re-observability"?

In abstract and introduction sections, the authors address the issue and proposed to focus on this issue. However, I cannot clearly understand the meaning of this terminology. From Line 51 to Line 53, my understanding is that the terminology indicates the ability to retrieve the target information from memory.

If so, I do not understand why the authors mentioned "existing scene representations, ..., lack post-hoc re-observability." (Line 13). I do not agree that 3DGS is the only representation can do "post-hoc re-observability". It is the matter of assigning semantic information or textual information on top of 3D primitives, not the inherent 3D representation problem.

Why language field is needed?

During the exploration, the authors collect many information like scene graphs, language fields, 3D Gaussian primitives, etc. I wonder why introducing language fields is needed. Simply VLM can be used as "VLM-as-Judge". I truly understand that VLMs do not produce the correct results, but the problem is that I do not understand the motivation of using language fields. Series of previous works like LangSplat, LeRF, etc target at the open-vocab 3D semantic segmentation task. It seems that this paper leverages the pipeline of previous methods. Then, ... I am quite lost at this point. According to Figure 2, I expect that authors want to claim both "object-level retrieval" and "semantic-level retrieval" are needed where the semantic-level retrieval requires language fields. With Table 2, I understand that both modules are requires to have higher performance, but to me, it is quite an engineering approach.

Questions for authors

Please resolve my worrying points addressed in the weakness section. Overall, this is kinda systemic paper that does not have much technical novelty or something insightful. While the authors addressed the usage of 3D Gaussians as 3D memory, but only partial module "semantic-level retrieval" is dependent on 3D Gaussians.

Reviewer_E1WYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GSMem, a framework that uses an incrementally updated 3D Gaussian map as spatial memory for embodied question answering and navigation. It combines object-level scene graphs and Gaussian language features to retrieve task-relevant regions, selects viewpoints for rendering, and uses the rendered observations for VLM reasoning. A hybrid exploration strategy combines semantic relevance with a Fisher-information-based geometric score. The framework is evaluated on OpenEQA and GOAT-Bench.

Critical strengths

‌1.‌The framework offers two complementary retrieval paths, one operating at the object level and the other at the semantic level. The semantic path is particularly useful because it allows a region to be retrieved even when the target is absent from the detected object nodes, which gives the system a fallback that purely object-based retrieval cannot provide.

‌2.‌The evaluation is carried out at the component level as well as the system level. The authors ablate retrieval, viewpoint selection, diffusion enhancement, and exploration, and they also compare different memory representations, viewpoint counts, and hyperparameter settings, so the reader can see how each design choice affects performance rather than only how the full pipeline behaves.

‌3.‌A physical robot demonstration is included in the supplementary material. It shows the mapping, retrieval, reasoning, and navigation components working together in an integrated system, which gives useful evidence that the approach is not confined to simulation.

Critical weaknesses

1.‌The motivation behind GSMem, and how it is positioned relative to prior Gaussian-based navigation methods, is not made clear enough. The Introduction motivates GSMem mainly through the re-rendering capability of 3DGS compared with scene graphs and image-based memories. However, GaussNav and 3DGSNav have also employed Gaussian maps and novel-view rendering for navigation, although their task settings differ. The remaining limitations that motivate GSMem's specific mechanisms are therefore not clearly identified relative to these works. The authors may clarify these limitations and the corresponding methodological differences in the Introduction, with a more detailed comparison in Related Work.

‌2.‌The relationship between the Fisher-trace score and information gain would also benefit from further justification. Section 3.4 defines geometric information gain through the log-determinant increase of the Fisher Information Matrix (Eq. 7), but replaces it with the trace of the incremental FIM (Eq. 8). This surrogate does not account for the current information matrix, so observations along already well-constrained directions may receive higher scores than those in less-constrained directions. Consequently, the score may not reflect the expected reduction in uncertainty. The authors may justify this approximation and compare it with coverage-based exploration scores or measured improvements in reconstruction quality.

‌3.‌The sampling protocol and the variability of the reported results are likewise not clearly specified. The component ablations in Table 2 use 10% of the GOAT-Bench validation-unseen split, yet neither the subset selection procedure nor the variability of the results is described. Small differences, such as the 1.2-percentage-point SR change from removing hybrid exploration, may depend on the sampled tasks. Reporting sample counts, subset composition, and variability across repeated evaluations would help assess these results.

Questions for authors

‌1.‌How does rendering quality at the selected viewpoints vary with prior observation coverage, and, compared with Random 6, what are the total runtime and VLM input costs, including candidate evaluation and opacity rendering?

‌2.‌What justifies replacing the log-determinant information gain with the trace of the incremental FIM in Eq. (8) without accounting for the current FIM, and how does this score correlate with actual coverage or reconstruction improvements?

‌3.‌How was the 10% GOAT-Bench validation-unseen subset in Table 2 selected, and how many scenes and subtasks does it contain? Are smaller performance differences, such as the 1.2-percentage-point SR change from removing hybrid exploration, consistent across repeated runs or different sampled subsets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.