acceptodds
Under review as a conference paper at ICLR 2027

Deep Exploration Through Pre-image State Distribution Matching (PIM)

Abstract

We present an algorithm for reinforcement learning that explores and learns to solve long-horizon, goal-reaching tasks without any environment rewards, demonstrations, curriculum, or prior knowledge. With our approach, a simulated robot can learn to construct complex 8-cube configurations, a task not previously solved by prior methods based on intrinsic motivation or subgoal selection. We show that the generalization capabilities of generative models trained from scratch can drive exploration, even in difficult, high-dimensional tasks. We train a generative model with only the online data collected by the policy to predict what states are likely to precede a goal state. Before the goal state been reached, this process is analogous to prompting an image generation model on an unseen prompt. Initially, this generative model makes poor predictions, but when the policy tries and fails to visit those states, it collects the data required to fix the model's predictions errors. Our algorithm outperforms a diverse set of baselines across multiple long-horizon tasks involving robotic manipulation, construction, and navigation. On the AllegroHand dexterous grasp-and-throw task, even the best-performing baseline does not obtain reliable success despite using more interactions.

Primary area:
reinforcement learning
Keywords:
exploration, goal-conditioned RL, state matching, reward shaping, Online RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_swdjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a new algorithm for RL that encourages exploration for long-horizon tasks. The approach trains a generative model online with the data collected by the policy which is then used to predict stages that are likely to lead to the goal state (goal state pre-image) and use this as an auxiliary reward to improve exploration. They find that their method is able to outperform a number of baselines on a number of long-horizon tasks.

Critical strengths

The main method, both the motivation for it, the mathematical formulation and description are all very good. The idea of training a generative model as a proxy signal to learning how to explore the environment has been done of course, but the paper really successfully walks through how the way they do this, by learning the distribution of states that may lead to the goal, is both different from prior work and very effective. The math is clear to follow and the right parts are included in the main paper versus the appendix, which is often a very difficult call to make.

The results themselves are very convincing. The number and quality of baselines compared with, the variety of environments where the method is shown to be effective and especially the visualization in Figure 3 all does a good job of convincingly showing that the method is not only effective, but is effective for the reason the authors say it is.

The quality of the presentation in general is very high. Again, this is a paper that would be quite difficult to understand is fairly readable. I put 2 for "Generally understandable, but requires substantial effort in places" but this is only because I think you do have to follow closely to understand it inherently. A more readable version of this paper would honestly be worse because it would not be trying to guide the reader through the math which I think would be a disservice.

Critical weaknesses

While I am generally pretty positive on the presentation, including of the main method, it is definitely hard to follow in places, although the authors have clearly done a heroic effort to explain quite a lot of math in the amount of space clearly. While I think they mostly succeed, I think there are a few places, especially in intro and abstract where things do not quite make sense yet on a first read.

Questions for authors

I honestly don't have many questions

1 comment

We thank the reviewer for the kind words and the high score. We have made some revisions to the introduction to simplify the wording and improve clarity. Please let us know if there are any particular sentences that you still find unclear.

Reviewer_UBvNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PreImage Matching (PIM), an exploration method that learns a conditional density model of future states from online experience. The log-likelihood assigned to a target goal is used as a reward for PPO, including before a successful trajectory to that goal has been observed. The method is interpreted as matching the policy's state occupancy to a dynamically evolving goal-preimage distribution. Experiments on long-horizon robotic construction, manipulation, and navigation tasks, including seven- and eight-cube construction, demonstrate substantial empirical potential. The paper also investigates density-model overfitting after the first success and introduces gradient regularization to improve stability. Producing directed exploration signals from experience without successful goal-reaching trajectories is an important research direction. However, the general improvement guarantee and the attribution of the empirical gains to the proposed mechanism are not yet fully established.

Critical strengths
  1. A promising approach to exploration before the goal has been reached. Sections 4-5 directly use the generalization of a generative model to an unseen goal to guide exploration. This offers a relatively simple research direction that differs from explicit novelty bonuses and addresses the important problem of obtaining useful directional signals without successful trajectories.

  2. Compelling difficult-task results and a useful analysis of instability. Figures 7-8 report results on multiple long-horizon tasks and with different density estimators. The seven- and eight-cube construction results are particularly interesting and provide substantial empirical motivation. Section 5.4 and Appendix C.6 further analyze degradation caused by density-model overfitting after the first success and show how state-gradient regularization improves stability. This failure-mode analysis adds practical value to the empirical contribution.

Critical weaknesses
  1. The connection between estimated preimage matching and actual goal-reaching improvement remains incomplete. Higher predicted goal likelihood need not imply greater actual success, especially before goal discovery. Appendix B.3 provides only a local gradient connection, while Appendix B.4 does not establish that its two-coordinate perturbation satisfies Bellman flow constraints. The omitted policy-dependent ELBO term and the unclear relation between pointwise density and success-region probability introduce further gaps.

  2. Empirical attribution and generality remain insufficiently established. Goal-assisted whitening and gradient regularizers are not matched across applicable baselines. The KL-based argument against Dirac-goal matching does not cover Wasserstein objectives. Experiments using structured states and high-level controllers also provide limited support for broader claims of learning without prior structure.

Questions for authors
  1. Can you construct a feasible descent direction satisfying the Bellman flow constraints in Appendix B.4 and test Theorem 1 on small MDPs with branches and mandatory intermediate states?

  2. Under the negative-binomial time weighting used in Appendix B.2, what valid improvement bound remains after dropping the first ELBO term? How does this relate to the geometrically discounted objective in Eq. (3)?

  3. Under what assumptions does increasing the learned goal log-likelihood improve actual goal-reaching performance? Please explain how the local first-order connection in Appendix B.3 extends to learned density models and practical PPO updates, and clarify the relationship between pointwise density and success-region probability. Targeted measurements of predicted likelihood versus actual success, including sensitivity to goal-noise scale, would help assess this connection.

  4. On the hardest construction tasks, can you separately ablate goal-assisted whitening and both gradient regularizers, and provide applicable exploration baselines with equivalent stabilization mechanisms to isolate the benefit of the preimage reward?

  5. For Wasserstein goal-distribution matching with state-distance transport costs, can you provide an applicable theoretical derivation or direct experiment rather than extending the KL-based conclusion?

11 comments

Thank you for the thoughtful feedback. It seems the main concerns are (1) whether estimated preimage matching provides a useful signal for goal-reaching, (2) the effect of gradient regularization and goal normalization on baselines, (3) the lack of Wasserstein-based baselines, and (4) theoretical gaps in Appendices B.3 and B.4.

To address these concerns, we provide clarifications below and have made the following additions:

  • (1-1) Training videos: We have attached training videos showing how the log-likelihood reward guides exploration toward goal-like states even before the first success.
  • (1-2) Role of out of distribution generalization: We have revised Section 4.1 and 6 to clarify the role of out-of-distribution generalization in the success of our method. We emphasize that the theoretical explanation for this phenomenon remains unknown and that neither our ELBO nor our policy improvement analysis explains it.
  • (2) Regularization and goal noise: We have added baseline experiments with smoothing regularization and goal normalization to investigate their effects on baseline performance.
  • (3) Wasserstein baselines: We are currently running Wasserstein-based baselines and will share the results as soon as they are available.
  • (4-1) Theoretical clarifications: We have added clarifications to Appendices B.3 and B.4 to address the theoretical concerns.
  • (4-2) Didactic experiment: Finally, we provide a didactic experiment illustrating monotonic improvement in goal-reaching probability as well as experiments for verifying the feasibility of constructing a descent direction, providing empirical support for Theorem 1.

All newly added content in the paper is highlighted in yellow.

We hope these additional explanations and evidence address the concerns. Please let us know if there are any other questions we can address.

1. Connection between estimated preimage matching and actual goal-reaching improvement

PIM matches the goal-preimage distribution even before the goal has been visited, and the main reason this works is the out-of-distribution (OOD) generalization capabilities of the generative model. The main contribution of our work is empirically showing that OOD generalization is sufficient to drive exploration even before reaching the goal across long-horizon tasks. The underlying theory of this OOD generalization phenomenon is still unknown to us. We have added clarifications in Sections 4.1 and 6 to make it clear that we do not claim to have a theory explaining this phenomenon.

The significance of our work is that we show that solving long-horizon tasks through online exploration does not require complicated subgoal selection, curriculum design, or novelty-seeking bonuses. Instead, the generalization capabilities of generative models are sufficient to drive goal-directed exploration. This is in sharp contrast to the dominant approaches to exploration in RL. In Figure 7, we show that our method, despite being simpler (no subgoal selection module, training only one policy for the task goal instaed of training a policy for all goals)than prior approaches, significantly outperforms them across all tasks. For instance, none of the baselines achieves stable success in the 7-cube and 8-cube tasks of builderBench and in Allegrohand. In Figure 5, we illustrate the conceptual advantage of our method over novelty-seeking approaches, which blindly attempt to explore novel states without any notion of goal-directedness. We show how these methods fail as the dimensionality of the problem increases (e.g., going from 3-cube to 4-cube stacking tasks), while our method can solve significantly higher-dimensional tasks without requiring additional exploration mechanisms. Furthermore, in Figure 8, we demonstrate that different choices of MC and TD density estimators exhibit the same exploration property, showing that this emergent exploration behavior is not specific to a particular density estimator but is universal across different density estimation methods.

Our visualizations in Section 5.1 and the Figure 1 illustrated how the estimated preimage distribution can provide a useful learning signal even before the goal is reached. To make this more concrete, we have uploaded videos of the AllegroHand task at different stages of training. These videos show how, even before the agent reaches the goal, the estimated goal log-likelihood provides a reward signal that gradually guides the agent toward the goal. Through trial and error, the agent eventually learns to complete the task. Importantly, this reward signal can also help the agent avoid actions that would cause it to get stuck, such as dropping the ball on the floor.

Neither the ELBO nor Theorem 1 explains this OOD generalization phenomenon. Instead, they address a different question: under an idealized setting where there is sufficient state coverage, does matching the goal-preimage distribution correspond to maximizing the probability of reaching the goal? The ELBO shows the connection between pγπ(g)p^\pi_\gamma(g) and the goal-preimage distribution, while Theorem 1 establishes this connection from a different perspective. Therefore, these results do not claim to explain why our method performs well at exploration. Instead, they serve as a sanity check that matching the goal-preimage distribution is a sensible objective for goal-reaching. Neither of these theoretical results is the main contribution of our work.

2. The generality of empirical attribution (Regularization and goal noise)

Goal-assisted whitening and gradient regularizers are not matched across applicable baselines.

We added the same state-smoothing regularizer used in PIM to our strongest baselines: RND, SGCRL, and DISCOVER. Specifically, we use the same primal-dual state-gradient penalty, augmenting the training objective with λ E[∥∇sf∥]\lambda\,\mathbb{E}[\lVert\nabla_s f\rVert], where f(s)f(s) is the predictor network for RND, the critic ϕ(s,a)⊤ψ(sf)\phi(s,a)^\top\psi(s_f) for SGCRL, and the goal-conditioned Q-value Q(s,a,sf)Q(s,a,s_f) for DISCOVER. The multiplier is updated as

λ←clip ⁣(λ+ηλ(E[∥∇sf∥]−c), 10−8, 1)\lambda\leftarrow\mathrm{clip}\!\left(\lambda+\eta_\lambda\bigl(\mathbb{E}[\lVert\nabla_s f\rVert]-c\bigr),\,10^{-8},\,1\right)

with target c=100c=100. For PPO+RND, the predictor fθ(s)∈RDf_\theta(s)\in\mathbb{R}^D is vector-valued, so we estimate ∥∇sf∥\lVert\nabla_s f\rVert using one Hutchinson probe, ∥Js(s)⊤ε∥/D\lVert J_s(s)^\top\varepsilon\rVert/\sqrt{D}, where ε∼N(0,I)\varepsilon\sim\mathcal{N}(0,I). We also used goal normalization for training the SGCRL and DISCOBVER critic. See the here.

We observe inconsistent results. Gradient regularization improves SGCRL's performance on the 4-cube stacking task, and adding goal noise improves it further, but these modifications still do not enable it to solve the harder 7- and 8-cube tasks. For RND, gradient regularization improves performance on the 3-cube task, but it still cannot solve the 4-cube task. For DISCOVER, we observe a degradation in performance on the 3-cube task, which is the easiest task, so we did not test it on harder tasks. Overall, it seems that these modifications help some baselines, but not enough to enable them to solve the hardest tasks.

Note that we only used gradient regularization for BuilderBench tasks, so we also limited these baseline experiments to BuilderBench. Here we only tested the best-performing baselines due to time limitations, but for the camera-ready version we will include results of other baselines too.

Experiments using structured states and high-level controllers also provide limited support for broader claims of learning without prior structure.

Note that the controller is a built-in feature only in BuilderBench and is neither a planner nor an exploration module. Its only role is to convert commands in the form of changes in xx, yy, and zz positions into forces that should be applied. Using such controllers is very common in RL to separate high-level planning from low-level control, particularly in a planning-heavy task such as block building. Moreover, we also evaluate our method on the 49-dimensional AllegroHand dexterous manipulation task, which does not use any controller and demonstrates the low-level control capabilities of our method while all other baselines fail to solve this task. Our MetaWorld peg insertion and maniskill don't use any controller as well.

3. Wasserstein baseline

We use (Durugkar et al.) as a baseline. We will share empirical results when prepared.

Reviewer_ME1cOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper "Deep Exploration Through Pre-image State Distribution Matching" proposes a new method for exploration in complex RL environements with sparse reward (authors say no reward, but this comes down to setting a binary reward when the goal state is reached). They propose to craft a dense reward for any state s, based on the estimated reachability of the goal g from s. This is a simple, but appealing, idea, which any many connexions with related work.

Critical strengths
  • Simple but appealing idea.
  • Good results
Critical weaknesses
  • The setting is unclear. Nothing is said about the behavior goal policy in training:

    • Do we keep the goal g always constant during training (this looks to be the case in section 5.1, but maybe not after) ? In that case, the good behavior observed in figure 3 is only possible because the maze is not too complex. This makes the assumption that the generative model will be always enough powerful to estimate reachability to the distant goal from any state of the maze, while that goal has never been seen at the begining of the process. To work, this requires that the representation space is smooth enough (this can explain the use of NF, which is limited to homeomorphisms, with smoothing regularization as the best generative approach), and that it well captures temporal distances related to the environement dynamics to hope getting good advices with such a mechanism. The maze in figure 3 is very simple and I supspect that more complex would lead the agent being stuck in deceptive dead ends: states that look close to the goal under the learned representation, yet are far from it, or cannot reach it at all, under the environment dynamics. In that case, the proposal fully relies on OOD generalization, which looks unfeasible in many settings. For that setting, selecting training goals from a replay buffer [1] or a model of admissible states [2,3,4] are possibilities, with priorization depending on the current competences of the agent.
    • Are we able to sample a goal everywhere in the state space ? This supposes that the state space is known a priori. In that setting, even without priorization of training goals, simply sampling goals from a uniform on the state space, even HER [5] can be able to solve hard tasks (though not the most efficient approach, but should be able to converge to a policy that solve the problem since the ability to sample everywhere allows acquire knowledge from the full environement dynamics). To make training more effective, many auto-tellic (or automatic curriculum) approaches have been developped : the main difficulty is how to prioritize training goals, based on novelty, learning progress, success entropy, etc. Authors do not say anything about this goal selection, which looks core for exploration in complex environements.
    • Are we in fully unknown exploration ? This is the most difficult setting. The objective is to discover new areas together with capabilities to go back to them. Approaches such as GoalGAN [7] (or Setter-Getter also using NF [8]) define an adversarial approach using a discriminator to assess wether any goal candidate is an admissible state with interest for learning, MEGA [1] samples from the replay buffer the goal candidates that are at the border of a known state distribution (with KDE), SVGG [2] uses stein particules as trackers of attention to discover not too hard and not too easy state areas, RIG [3] train a VAE on reached states and sample imagined goals from the induced distribution. SkewFit [6] and ImagineBeyond [4] deal with distributionnaly robust optimization to stear the VAE distribution toward less known states, etc.
  • Innovation of the proposal and positioning: beyond connexions to all the litterature that I mentionned above, I feel there are also connexions with hierarchical approaches that should be considered. At least, the paper [9] selects at each step on the way of the final goal a subgoal that should be reached before, given an advantage estimation. Regarding training without reward, which is one claim of the paper, authors should discuss Forward-Backward approaches [10], whose objective look very similar to the one of the paper.

  • Introduction claims that the method scale for high-dimentional states. I do not understand that claim since the method directly works with states. What kind if high-dimentional inputs has been considered ? 23 as I see in the xp section ? Is it possible to deal with images as instance ?

  • Writting is rendered overcomplicated in section 4.1, while the Elbo and all the approach is quite simple. the section says KL , then no finally general, then finally reverse KL, then we could also with forward KL, etc. Also the sentence "Applying the methods of lagrangian shows that the optimal q(s) is the goal preimage distribution" looks imprecise, as it is well known and direct to show that : ∫sp(s∣g)log⁡p(g∣s)p(s)p(s∣q)ds=∫sp(s∣g)log⁡p(g)ds=log⁡p(g)∫sp(s∣g)ds=log⁡p(g)\int_s p(s|g) \log \frac{p(g|s)p(s)}{p(s|q)} ds= \int_s p(s|g) \log p(g) ds= \log p(g) \int_s p(s|g) ds = \log p(g), hence q(s)=p(s∣g)q(s)=p(s|g) is the optimum. Please simplify the section, being more direct.

[1] Mega : S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba. Maximum entropy gain exploration for long horizonmulti-goal reinforcement learning. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7750–7761. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/ v119/pitis20a.html.

[2] SVGG : N. Castanet, S. Lamprier, and O. Sigaud. Stein variational goal generation for adaptive exploration in multi-goal reinforcement learning. arXiv preprint arXiv:2206.06719, 2022.

[3] RIG : A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807.04742, 2018.

[4] Imagine Beyond : Nicolas Castanet, Olivier Sigaud, Sylvain Lamprier: Imagine Beyond ! Distributionally Robust Autoencoding for State Space Coverage in Online Reinforcement Learning. NeurIPS 2025

[5] HER: Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., ... & Zaremba, W. (2017). Hindsight experience replay. Advances in neural information processing systems, 30.

[6] SkewFit : V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.

[7] GoalGAN : C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515–1528. PMLR, 2018.

[8] Setter-Getter : Racanière, Lampinen, Santoro, Reichert, Firoiu, Lillicrap, « Automated Curriculum Generation Through Setter-Solver Interactions », ICLR 2020

[9] Pi-switch : Stojanovic, S., & Proutiere, A. (2026). Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning. arXiv preprint arXiv:2605.13207.

[10] Forward Backward : Touati, A., & Ollivier, Y. (2021). Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34, 13-23.

Questions for authors

Please see weaknesses for my questions. Beyond answering questons, please consider discuss the setting and the position regarding the references I mention.

Minor:

  • (8) could better emphasize the use of NF. I would advice to use (33) in place, which is greatly more informative.

Comments on the paper

4 comments

We thank the reviewer for their response. We have further clarified the significance of our results relative to prior work. We are also extending our experiments to a pixel-based observation Kinetix task [8] and the Unitree H1 humanoid task. Our preliminary results are encouraging, and we plan to share the completed results soon.

The method is standard RL with a learned reward, the log-probability of reaching from, and such reachability-based rewards already exist

We agree that using learned reachability-based rewards in RL is not new. Our main contribution is showing that a generative model trained entirely from scratch and without any subgoal selection can provide a sufficiently informative reward to drive exploration in challenging long-horizon tasks, even before the agent has reached the goal, and training one single hard goal-reaching policy and value function is enough (as opposed to training policies for all goals via HER)

DDL [2] heavily relies on selecting intermediate goals from previously visited states for exploration and learning to reach them as opposed to learning and maximizing one reward only for the hard task. Moreover, its empirical results are considerably limited compared to ours. Our method solves much harder long-horizon tasks, including eight-cube tower construction and dexterous manipulation.

ODAC [1] is closer to our setting, but it uses a one-step dynamics model as opposed to a discounted occupancy model as the shaped reward. Moreover, they train a goal-conditioned policy and value function across many hindsight-relabeled goals using HER and the method highly relies on that, whereas our method trains a single PPO policy and value for one fixed goal, without any hindsight relabeling in the actor or critic, making our approach considerably simpler. Despite being more complicated, ODAC succeeds on relatively simple pushing/pulling tasks and Ant locomotion, whereas our method solves much more challenging long-horizon tasks that require planning, such as learning to use blocks as scaffolding in the seven-cube task (Fig. 6). Plus, In Section 5.4 (l 510), we directly compare against a one-step likelihood reward and show that it only solves the easier tasks, highlighting the importance of modeling discounted future-state occupancy rather than one-step transitions. We have clarified these differences in Section 2 of the paper.

Method only works where state-space proximity reflects reachability

Our learned reward is substantially more informative than a simple Euclidean distance to the goal. In the past, we evaluated an L2 distance reward on the three- and four-cube stacking task as a sanity check, and it failed to achieve reliable success. Now we are reproducing those baseline results, and we will share the results. The BuilderBench paper [4] also demonstrates the limitations of L2 distance-based rewards for block-building tasks (Fig. 11, top row, PPO baseline).

The difficulty of reward design is even more apparent in dexterous manipulation. Prior work with AllegroHand, such as DexArt [6], explicitly designs multi-stage rewards for reaching, establishing contact, and manipulating objects, along with additional regularization terms. Other work, such as Eureka [7], specifically studies the difficulty of designing effective rewards for dexterous manipulation. Our method instead learns a shaped reward from scratch that enables a dexterous hand to grasp an object and throw it into a bucket. We believe that replacing this labor-intensive, manually designed reward engineering process with a reward that can be learned autonomously from scratch is itself a significant contribution. If the reviewer considers it necessary, we can also run a direct L2 distance reward baseline for all of our tasks to further validate this distinction.

References

[1] Rudner, T. G. J., et al. (2021). Outcome-Driven Reinforcement Learning via Variational Inference. NeurIPS.

[2] Hartikainen, K., Geng, X., Haarnoja, T., & Levine, S. (2019). Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery. arXiv:1907.08225.

[4] Ghugare, R., et al. (2026). BuilderBench: The Building Blocks of Intelligent Agents. arXiv:2510.06288.

[6] Bao, C., Xu, H., Qin, Y., & Wang, X. (2023). DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects. CVPR.

[7] Ma, Y. J., et al. (2024). Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR.

[8] Matthews et al. (2025). Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks

Thanks for increasing your score and for clarifying your concerns. Thank you for engaging with us during the rebuttal period and for the thoughtful discussion.

the method won't work where target states often lie in regions very different from those reached early on, whose dynamics seem impossible to predict without ever visiting them

We agree that there is a limit to the performance of our method and that it cannot solve ANY complex task. Further analysis to characterize the tasks that our method cannot solve is left for future work. However, to be fair, one would not have expected our method to solve complex tasks such as the 8-cube task either. Therefore, it is hard to speculate about the limitations of this method without empirical evidence, especially since science is often about discovering surprising results that might not seem plausible at first glance.

The findings are too counter-intuitive to me

We find the unintuitive nature of our empirical results to be exactly the main significance of this work. A simple method that people would not expect to work, and therefore shifted their focus to much more complicated subgoal selection and curriculum methods necessary, does indeed work. It is commonly believed that generative models need large amounts of diverse data to generalize well, but our results challenge this belief by showing that the generalization capabilities of neural networks can be very useful even with limited data, and can alone guide the collection of new data, closing the loop between learning and data collection.

We believe some of the most impactful research results come from showing that a surprisingly simple idea can do something people did not think it could do. Our work suggests a shift in exploration research: instead of focusing on designing increasingly complicated subgoal selection methods, researchers should focus on improving the architecture and regularization of generative models to achieve better generalization and therefore better exploration. And it is yet an open problem to discover the limits of this method.

Similarity to Single goal contrastive RL

We understand your concern about the similarity to SGCRL. However, apart from significant empirical superiority, there are three important conceptual differences. First, even if we only care about solving one hard task, SGCRL still requires training policies for all goals in an off-policy manner. This might sound unimportant, but apart from being computationally less efficient makes it difficult to apply SGCRL to standard RL pipelines, where we typically have a policy trained with PPO/SAC to maximize some reward. Our method can be directly plugged into this standard reward-maximization pipeline by simply replacing the reward with our online shaped reward, whereas SGCRL cannot. Our method does reward shaping; therefore, it can easily integrate sparse environment rewards if available, while SGCRL doesn't. Second, SGCRL leaves an important question unanswered: Is the density estimator alone sufficient for solving long-horizon tasks, or do we also need to train policies for all goals? Our work shows that the density estimator alone can provide a sufficient exploration signal to solve these tasks. Indeed, we still sample states from a replay buffer to train the density estimator, but this seems to be the minimum required ingredient to solve such a problem.

Third, prior work [1] attributes the success of SGCRL to the particular geometry of contrastive representations and suggests that this emergent exploration phenomenon is unique to contrastive RL. Our work challenges this claim by showing that the same exploration phenomenon can be reproduced and even significantly improved using a variety of density estimators, including normalizing flows, UVD, and TD-InfoNCE (Fig. 8). We believe this is an important distinction, as it shows that the phenomenon is much more general than previously thought. These comparisons can be found in line 144 of the revised paper.

We will share our new empirical results soon.

[1] bastankhah et al., 2026. Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL

AuthorsOpenReview ↗

New Empirical Results (part 2)

l2 distance baseline

PIM successfully solves all the tasks below within the same training budget, often in substantially fewer steps.

TaskSuccessSteps
Allegro throw cube0.042500M
Unitree H1 humanoid0.006500M
Maze 21×210.000132M
BuilderBench 3-cube stack0.005100M
BuilderBench 4-cube stack0.007200M
BuilderBench 5-cube pyramid0.000200M
BuilderBench 5-cube overhang0.000200M
BuilderBench 7-cube zig-zag0.000164M
BuilderBench 8-cube Jenga0.000126M

We observed that a common failure mode of the L2-distance baseline is that it greedily matches parts of the state to the goal but fails to complete the full task. For example, in this video of the pyramid-building task, the agent manages to place some cubes very close to their target positions but fails to construct the complete structure.

Is PIM empirically superior to prior HER-based methods such as SGCRL despite its simpler construction? Does it outperform an L2-distance reward baseline?

We believe our newly added maze results provide further evidence of PIM's superiority over HER-based methods such as SGCRL and reward-shaping approaches such as the L2-distance baseline, consistent with our findings in the other environments in the paper. Moreover, PIM can solve the humanoid task where an L2-distance-based method cannot.

We will add more baseline experiments for these newly added tasks in the camera-ready version of the paper.

AuthorsOpenReview ↗

Summary of the paper and the discussion for AC

Dear AC,

To help aid in the final decision, we provide a summary of our work and updates we made following the review discussion.

Our work introduces an algorithm for exploration in long-horizon, sparse-reward tasks that relies on the generalization capabilities of generative models trained from scratch. Our method theoretically corresponds to estimating the distribution of states visited before reaching the goal and training a policy to match this distribution. We demonstrate significant improvements over prior exploration methods, including novelty-based, hindsight experience replay (HER), and subgoal-based approaches, solving challenging 7- and 8-cube tower building tasks where none of the baselines achieves a single success as well as high-dimensional dexterous manipulation and humanoid tasks. The significance of our paper is that it shows that a single, dynamically updated reward can solve exploration tasks that all elaborate existing methods fail to solve, potentially shifting exploration research toward leveraging generative models' generalization rather than designing increasingly complex subgoal selection and curriculum strategies or novelty seeking methods.

The main concerns of reviewer ME1c were (i) whether our method generalizes to environments with unpredictable dynamics where generalization to faraway regions is hard, (ii) the lack of an L2-distance reward baseline, (iii) they found the fact that our method relies on out-of-distribution (OOD) generalization of generative models unintuitive, and (iv) the lack of high-dimensional experiments. To address these concerns, we (i) added a challenging maze environment with arbitrary walls, where dynamics are hard to predict, (ii) added L2-distance baselines across all tasks, (iii) clarified that our main contribution is showing that a seemingly unlikely approach works remarkably well, and (iv) added two pixel-based tasks and a high-dimensional Unitree H1 humanoid task.

The main concerns of reviewer UBvN were (i) a missing assumption in a theoretical result in the appendix, which is not part of our main contribution, (ii) the lack of Wasserstein-based goal distribution matching baselines, and (iii) unclear use of the sparse environment reward, which is only nonzero at the goal. To address these concerns, we (i) added the missing assumption to the appendix, (ii) implemented and evaluated the requested Wasserstein-based baseline, and (iii) clarified that the sparse reward is only used in harder tasks to stabilize success after reaching the goal, rather than to drive exploration, and that the same reward is also provided to applicable baselines.

All newly added clarifications in the paper are highlighted in yellow, and the new baseline results will be added to the camera-ready version.

Kind Regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.