Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Exploration is a prerequisite for learning useful behaviors in sparse-reward, long-horizon tasks, particularly in 3D environments. Curiosity-driven reinforcement learning addresses this via intrinsic rewards derived from the mismatch between the agent's predictive model of the world and reality. However, translating this intrinsic motivation to complex, photorealistic environments remains difficult, as agents can become trapped in local loops and receive fresh rewards for revisiting forgotten states. We show that this failure of prediction-error curiosity stems from a lack of spatial persistence and episodic context. Effective curiosity requires a model of the world that is persistent and continuously updated, paired with an agent that maintains an episodic trajectory history to navigate toward novel regions. We achieve this using an online 3D reconstruction as a persistent model of the world, while the agent policy is parameterized as a sequence model over RGB observations to maintain episodic context. This design enables effective exploration during training while allowing the agent to navigate using solely RGB frames at deployment. Trained purely via curiosity on HM3D, our deployed agent effectively explores 3D scenes using only RGB input. We validate the strength of this curiosity-driven behavior by showing that it outperforms specialized active-mapping policies with matching input and generalizes zero-shot to Gibson and AI-generated worlds. Further, our end-to-end policy adapts efficiently to downstream tasks such as apple picking and image-goal navigation, demonstrating the flexibility and transferability of behavior that emerges from curiosity in 3D worlds.