acceptodds
Withdrawn from ICLR 2027

Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploration is a prerequisite for learning useful behaviors in sparse-reward, long-horizon tasks, particularly in 3D environments. Curiosity-driven reinforcement learning addresses this via intrinsic rewards derived from the mismatch between the agent's predictive model of the world and reality. However, translating this intrinsic motivation to complex, photorealistic environments remains difficult, as agents can become trapped in local loops and receive fresh rewards for revisiting forgotten states. We show that this failure of prediction-error curiosity stems from a lack of spatial persistence and episodic context. Effective curiosity requires a model of the world that is persistent and continuously updated, paired with an agent that maintains an episodic trajectory history to navigate toward novel regions. We achieve this using an online 3D reconstruction as a persistent model of the world, while the agent policy is parameterized as a sequence model over RGB observations to maintain episodic context. This design enables effective exploration during training while allowing the agent to navigate using solely RGB frames at deployment. Trained purely via curiosity on HM3D, our deployed agent effectively explores 3D scenes using only RGB input. We validate the strength of this curiosity-driven behavior by showing that it outperforms specialized active-mapping policies with matching input and generalizes zero-shot to Gibson and AI-generated worlds. Further, our end-to-end policy adapts efficiently to downstream tasks such as apple picking and image-goal navigation, demonstrating the flexibility and transferability of behavior that emerges from curiosity in 3D worlds.

Primary area:
reinforcement learning
Keywords:
Curiosity-driven RL, 3D Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Go6BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper train an RGB-only exploration agent in Habitat using what they called curiosity. During their training, the reward is taken from the persistent 3D GS-reconstruction built with depth and pose. The agent gets rewarded for views the reconstruction simply cannot predict. The policy is a transformer over past frames, so at test time it needs no depth, pose, or map. In results, it beats RGB-based ANS and OccAnt, and fine-tunes well on apple picking and image-goal navigation.

Critical strengths

Using a privileged reward model only in training and dropping it at deployment is simple and sounds practical. Matching depth-based OccAnt without any depth at test time is a genuinely useful result, and the memory ablations are clean.

Critical weaknesses

The reward is basically a coverage reward computed with ground truth depth and pose. The reconstruction is near perfect on seen regions, so “curiosity” here just means “go somewhere new.” The paper never compares against a simple new-voxel reward or standard novelty baselines like RND or E3B, so there’s no evidence that the 3DGS formulation is doing anything the simpler version wouldn’t.

The evidence might be not complete enough. There’s one training, the gap to OccAnt-RGBD is 0.3 points, and without any fine-tuning the agent already hits 48% on image-goal navigation. That number suggests the downstream task might mostly just rewards coverage.

Questions for authors
  • How does your reward compare with a simple coverage reward from the same depth and pose, and with RND or E3B?
  • Does image-goal success require the agent to stop?
  • Can you report a few training seeds for the main results?
Reviewer_ezQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a curiosity-driven exploration framework in photorealistic 3D environments, emphasizing persistent world modeling and episodic policy memory. During training, an online 3D Gaussian Splatting reconstruction uses privileged depth and pose to provide intrinsic rewards based on visual prediction error. A Transformer policy with temporal attention and recurrent memory learns exploratory behavior and operates using only RGB observations and action history at deployment. Experiments evaluate exploration on HM3D, zero-shot transfer to Gibson and AI-generated worlds, and adaptation to apple-picking and image-goal navigation tasks.

Critical strengths

The paper’s main strength is its ability to learn transferable exploration behavior through a curiosity-driven Transformer policy. Compared with explicit active mapping approaches, the policy operates directly from RGB observations and action history at deployment, while persistent world modeling during training and episodic policy memory help reduce repetitive revisiting and encourage exploration of unseen regions. The results on Gibson and the qualitative demonstrations in AI-generated worlds provide encouraging evidence of transfer beyond the training environments. This generalization ability is particularly consequential to my assessment, as it suggests that curiosity-based pretraining can support useful exploratory behavior when external rewards are sparse.

Critical weaknesses

The main weakness is that the current evaluation does not clearly demonstrate the advantages of the learned Transformer policy. If completeness and coverage are the primary criteria, the improvements over OccAnt are limited: Appendix H, Table 3 reports nearly identical HM3D completeness at 1,024 steps for the proposed method and OccAnt-RGBD (74.69% versus 74.42%), with overlapping one-standard-deviation intervals across evaluation seeds. Achieving this performance with RGB-only observations is valuable, but the coverage advantage remains modest. Since the method learns a navigation policy, its benefits should also be assessed through navigation efficiency, including path length and redundant revisiting during exploration, and SPL for the downstream image-goal navigation task. The absence of these results leaves unclear whether comparable coverage is achieved through more efficient behavior. Furthermore, scene-specific active mapping approaches such as ActiveGAMER and ActiveSplat avoid costly RL policy training, while the proposed method offers a reusable policy that transfers across scenes. Without comparisons against these approaches, the trade-offs among training cost, transferability, completeness, and navigation efficiency remain insufficiently characterized.

Questions for authors
  1. Figure 4 illustrates baseline failure cases, yet the overall completeness improvement over OccAnt is modest, particularly in Appendix H, Table 3. Could the authors explain this discrepancy? Does the proposed policy exhibit other failure modes that offset its advantages? An analysis of existing evaluation trajectories would help clarify when and why it outperforms or underperforms the baselines.
  2. Since the paper pick OccA and ANS as the baselines, and both focus on exploration, the navigation efficiency during exploration is also important. Are navigation-efficiency results available for the proposed method, OccAnt, and ANS under comparable settings? In particular, SPL alongside success rate on the downstream image-goal navigation task would help determine whether the learned policy offers benefits beyond coverage. For exploration, path length or repeated visitation would provide complementary evidence. If these measurements are unavailable, please clarify the scope of the current evaluation.
Reviewer_b5KuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem that forward-prediction-error curiosity mechanisms in 3D embodied navigation tend to fall into looping traps in previously visited regions, and proposes an asymmetric curiosity exploration framework. During training, it uses privileged depth and ground-truth poses to online-fit a 3D Gaussian Splatting model, and adopts the error of Gaussian-rendered blurred images as the exploration reward. During inference, the 3D model is entirely discarded, and exploration relies solely on a lightweight RGB transformer policy composed of sliding-window self-attention and global linear attention for pure visual exploration. Zero-shot transfer and fine-tuning evaluations on real indoor scene datasets show that the method effectively improves visual coverage and mapping efficiency.

Critical strengths
  • The paper points out that visual forward-prediction-error-driven exploration in indoor environments repeatedly generates spurious novelty rewards due to slight viewpoint changes or complex textures, and proposes replacing the conventional single-step forward prediction error with global 3D reconstruction uncertainty.
  • During training, it leverages the efficient fitting capability of 3D Gaussian Splatting to aggregate global spatial memory; during inference, it entirely removes depth and 3D point-cloud overhead and runs only a pure vision policy, making it suitable for resource-constrained onboard edge devices.
Critical weaknesses
  • The empirical validation claims substantial improvements over existing exploration approaches, but the selected baselines are severely outdated. While the paper positions 3DGS as a central novelty for world modeling, it entirely omits comparisons with published neural/Gaussian active mapping baselines.
  • The curiosity baselines are evaluated using small encoders. In stark contrast, the proposed policy enjoys a modern DINOv2 foundation-model backbone, Plücker ray directional embeddings, and a dual-attention transformer architecture. The paper does not provide an architecture-matched control. Without benchmarking against contemporary embodied exploration methods and isolating backbone advantages, the empirical superiority of the proposed 3DGS curiosity reward remains scientifically unvalidated.
Questions for authors
  • Could the authors provide architecture-matched baselines by either upgrading standard curiosity methods with the proposed DINOv2 dual-attention backbone or testing the 3DGS reward on ResNet-18?
  • Could the authors clarify how the proposed method compares against contemporary active 3DGS/NeRF exploration baselines under equivalent training budgets?
  • Could the authors provide a direct comparison against a privileged ground-truth voxel frontier coverage reward to justify the high computational cost of online 3DGS blurred rendering?
  • Could the authors validate sim-to-real transfer by deploying the policy on a physical mobile robot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.