-
The setting is unclear. Nothing is said about the behavior goal policy in training:
- Do we keep the goal g always constant during training (this looks to be the case in section 5.1, but maybe not after) ? In that case, the good behavior observed in figure 3 is only possible because the maze is not too complex. This makes the assumption that the generative model will be always enough powerful to estimate reachability to the distant goal from any state of the maze, while that goal has never been seen at the begining of the process. To work, this requires that the representation space is smooth enough (this can explain the use of NF, which is limited to homeomorphisms, with smoothing regularization as the best generative approach), and that it well captures temporal distances related to the environement dynamics to hope getting good advices with such a mechanism. The maze in figure 3 is very simple and I supspect that more complex would lead the agent being stuck in deceptive dead ends: states that look close to the goal under the learned representation, yet are far from it, or cannot reach it at all, under the environment dynamics. In that case, the proposal fully relies on OOD generalization, which looks unfeasible in many settings. For that setting, selecting training goals from a replay buffer [1] or a model of admissible states [2,3,4] are possibilities, with priorization depending on the current competences of the agent.
- Are we able to sample a goal everywhere in the state space ? This supposes that the state space is known a priori. In that setting, even without priorization of training goals, simply sampling goals from a uniform on the state space, even HER [5] can be able to solve hard tasks (though not the most efficient approach, but should be able to converge to a policy that solve the problem since the ability to sample everywhere allows acquire knowledge from the full environement dynamics). To make training more effective, many auto-tellic (or automatic curriculum) approaches have been developped : the main difficulty is how to prioritize training goals, based on novelty, learning progress, success entropy, etc. Authors do not say anything about this goal selection, which looks core for exploration in complex environements.
- Are we in fully unknown exploration ? This is the most difficult setting. The objective is to discover new areas together with capabilities to go back to them. Approaches such as GoalGAN [7] (or Setter-Getter also using NF [8]) define an adversarial approach using a discriminator to assess wether any goal candidate is an admissible state with interest for learning, MEGA [1] samples from the replay buffer the goal candidates that are at the border of a known state distribution (with KDE), SVGG [2] uses stein particules as trackers of attention to discover not too hard and not too easy state areas, RIG [3] train a VAE on reached states and sample imagined goals from the induced distribution. SkewFit [6] and ImagineBeyond [4] deal with distributionnaly robust optimization to stear the VAE distribution toward less known states, etc.
-
Innovation of the proposal and positioning: beyond connexions to all the litterature that I mentionned above, I feel there are also connexions with hierarchical approaches that should be considered. At least, the paper [9] selects at each step on the way of the final goal a subgoal that should be reached before, given an advantage estimation. Regarding training without reward, which is one claim of the paper, authors should discuss Forward-Backward approaches [10], whose objective look very similar to the one of the paper.
-
Introduction claims that the method scale for high-dimentional states. I do not understand that claim since the method directly works with states. What kind if high-dimentional inputs has been considered ? 23 as I see in the xp section ? Is it possible to deal with images as instance ?
-
Writting is rendered overcomplicated in section 4.1, while the Elbo and all the approach is quite simple. the section says KL , then no finally general, then finally reverse KL, then we could also with forward KL, etc. Also the sentence "Applying the methods of lagrangian shows that the optimal q(s) is the goal preimage distribution" looks imprecise, as it is well known and direct to show that : ∫sp(s∣g)logp(s∣q)p(g∣s)p(s)ds=∫sp(s∣g)logp(g)ds=logp(g)∫sp(s∣g)ds=logp(g), hence q(s)=p(s∣g) is the optimum. Please simplify the section, being more direct.
[1] Mega : S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba. Maximum entropy gain exploration for long horizonmulti-goal reinforcement learning. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7750–7761. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/
v119/pitis20a.html.
[2] SVGG : N. Castanet, S. Lamprier, and O. Sigaud. Stein variational goal generation for adaptive exploration in multi-goal reinforcement learning. arXiv preprint arXiv:2206.06719, 2022.
[3] RIG : A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with
imagined goals. arXiv preprint arXiv:1807.04742, 2018.
[4] Imagine Beyond : Nicolas Castanet, Olivier Sigaud, Sylvain Lamprier: Imagine Beyond ! Distributionally Robust Autoencoding for State Space Coverage in Online Reinforcement Learning. NeurIPS 2025
[5] HER: Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., ... & Zaremba, W. (2017). Hindsight experience replay. Advances in neural information processing systems, 30.
[6] SkewFit : V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised
reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.
[7] GoalGAN : C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning
agents. In International conference on machine learning, pages 1515–1528. PMLR, 2018.
[8] Setter-Getter : Racanière, Lampinen, Santoro, Reichert, Firoiu, Lillicrap, « Automated Curriculum Generation Through Setter-Solver Interactions », ICLR 2020
[9] Pi-switch : Stojanovic, S., & Proutiere, A. (2026). Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning. arXiv preprint arXiv:2605.13207.
[10] Forward Backward : Touati, A., & Ollivier, Y. (2021). Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34, 13-23.
We thank the reviewer for the kind words and the high score. We have made some revisions to the introduction to simplify the wording and improve clarity. Please let us know if there are any particular sentences that you still find unclear.