acceptodds
Under review as a conference paper at ICLR 2027

Jumpy Planning over Policy Occupancies Improves Online Goal-Conditioned RL

24 views

Abstract

Online goal-conditioned reinforcement learning aims to learn a single policy for many goals, but sparse feedback makes long-horizon behaviors difficult to discover from interaction alone. Planning with temporal abstractions through Geometric Policy Composition could accelerate the acquisition of such behaviors but existing methods have largely focused on fixed, pretrained policies and world models. We introduce Online Geometric Policy Composition (OGPC), which plans over short subgoal sequences to guide exploration while learning both the policy and its world model online from scratch. OGPC uses a Geometric Horizon Model to predict which states the policy reaches while pursuing each subgoal, and chains these predictions to evaluate candidate sequences. To score candidate plans reliably under sparse rewards, we replace the final sampled reward in the existing Monte Carlo estimator with a learned estimate of its conditional success probability. With this estimator, OGPC improves sample efficiency and final performance over a strong, depth-scaled Contrastive Reinforcement Learning policy across nine JaxGCRL tasks, including high-dimensional Humanoid environments. The gains persist without planning at evaluation, showing that temporally abstract planning improves the learned policy rather than only decision-time behavior.

Primary area:
reinforcement learning
Keywords:
online goal-conditioned reinforcement learning, jumpy world model, successor measure, policy occupancy, geometric policy composition

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_4JiJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper proposes an extension of Jumpy World Planning with occupancy models to the online estimator and proposes an estimator for the terminal component of the occupancy density under a geometric switching policy. The results indicate that the terminal estimator reduces variance and produces significantly better performance in some cases.

Critical strengths

The major strength is that the paper rigorously shows that using a conditional density estimator for the terminal component reduces variance and leads to better performance than using a count-based density estimator.

Critical weaknesses

The paper seems like a direct extension of Jumpy World Models (Farebrother et. al., 2026) to online RL with new a terminal component density estimator and a different planning mechanism (random shooting vs iCEM planner). The proposed method to learn this estimator does not seem to significantly outperform the MC version in 5/9 environments (Figs 4 and 5).

Questions for authors

1 and 2. The density estimator's labels seem to come from MC sampling. I'm guessing that the labels are 0 or close to 0 far away from goal. The density estimator would still produce very small values in such a scenario. Why not use a different density estimator such as CRL/Hutchinson Trace Estimator since you're using a flow-matching model? Following up on this, isn't it technically possible to use density estimators for all components rather than just the last one? Do you have experiments that test this out?

  1. Could you clarify how many future states are sampled when running the MC version in the experiments section?

  2. Minor comment: "4.2 Q2: DOES OGPC EXTEND TO HIGH-DIMENSIONAL CONTROL?". Please clarify that this refer to a high-dimensional action space.

3 comments

We thank the reviewer for their constructive feedback and for taking the time to review our work. We would like to clarify how OGPC extends prior work and where terminal conditioning provides additional benefits.

W1. The paper seems like a direct extension of Jumpy World Models (Farebrother et. al., 2026) to online RL

Relationship to Jumpy World Models: Our work indeed builds on Farebrother et al., 2026 [2], which itself builds on GPC [4]. However, we would like to remark that Farebrother et al. explicitly identify “jointly learning policies and predictive models” as a promising future direction in their Discussion (Section 5). The online setting, where the policy and its GHM are trained concurrently from scratch, was left open by their work.

W2. The proposed method to learn this estimator does not seem to significantly outperform the MC version in 5/9 environments (Figs 4 and 5)

We would like to add some context to the significance of the result: OGPC-TC demonstrates most significant improvements over OGPC-MC on Ant Big Maze, Ant Hardest Maze, Humanoid U-Maze, and Humanoid Big Maze. They are among the most challenging tasks in JaxGCRL [1]. On final Time at Goal (Tables 19 and 20, Appendix F.1), TC has 95% confidence intervals that lie entirely above those of MC on six out of nine tasks. Moreover, TC demonstrates faster learning than MC on Humanoid. We believe that these results show benefits in both performance on challenging tasks and sample efficiency.

The density estimator's labels seem to come from MC sampling. I'm guessing that the labels are 0 or close to 0 far away from goal. The density estimator would still produce very small values in such a scenario.

Thank you for raising this point. You are correct that the terminal head's targets are Monte Carlo estimates, and that these targets can be zero when goal occupancy is rare. In our setting, small probabilities are appropriate for unlikely outcomes. The benefit is to preserve meaningful differences between candidate plans that would otherwise receive identical zero scores from finite-budget MC scoring.

Q1. Why not use a different density estimator such as CRL / Hutchinson Trace Estimator since you’re using a flow-matching model?

Thank you for suggesting these alternatives. We use a terminal conditional head because plan scoring requires repeated estimates of the conditional probability mass assigned to the task’s goal region,

q(s,a,gK,g⋆,γ)=m^γπgK(Xg⋆∣s,a)q(s,a,g_K,g^\star,\gamma) =\widehat m_\gamma^{\pi_{g_K}}(X_{g^\star}\mid s,a)

and the head provides this quantity directly at low computational cost during planning.

Hutchinson Trace Estimator is impractical in our online setting: A Hutchinson-based approach could evaluate the flow model’s log-density through the instantaneous change of variables formula [3]. However, this requires integrating the flow together with Jacobian-trace estimates, and the resulting state density would still need to be integrated over the goal region, including marginalization over state coordinates that do not define success. At our planner's settings (48 candidates, 64 MC samples per hop, 5 Euler steps), an importance-sampled flow-likelihood estimate of the goal region mass with 64 integration points requires 640 GHM evaluations for the terminal term of every simulation, compared to 5 for the Monte Carlo estimator and none for the terminal head. Instead, the terminal head amortizes the cost of this goal-region probability estimation across training updates and makes it directly available during planning.

The existing CRL critic is not directly interchangeable as the terminal conditional head: We acknowledge a contrastive estimator is also a possible alternative. However, the existing CRL critic is not directly interchangeable with our head as the contrastive score represents a density ratio rather than the finite goal-region occupancy. Moreover, the CRL critic takes a single goal as input, whereas our TC head evaluates the policy commanded to gKg_K at a task goal g⋆g^\star. A suitable contrastive estimator could be developed, and we believe that it would constitute an interesting extension by itself. However, this falls outside the scope of this paper.

The exact terminal-conditioning result is independent of how the conditional probability is computed, and the experiments establish the usefulness of a simple learned approximation compared with literal MC scoring. We do not claim to establish superiority over flow-likelihood or contrastive alternatives. We will clarify this rationale for the head in Section 3.2.

Isn’t it technically possible to use density estimators for all components rather than just the last one? Do you have experiments that test this out?

Thank you for raising this point. Following your question, we implemented a variant in which every component of the score uses a learned conditional probability (we call this variant “all-component”). Here’s the comparison between the terminal-only OGPC-TC runs against the “all-component” variant on Ant Big Maze and Pusher Easy tasks:

TaskMetricAll-component - OGPC-TC
Ant Big MazeTime at Goal-39.0 [-101.4, +23.4]
Success Rate-4.3% [-13.5, +4.9]
Pusher EasyTime at Goal-54.0 [-129.5, +21.5]
Success Rate-1.0% [-3.6, +1.5]

We found that estimating the conditional probability at every hop performed worse than OGPC-TC on average across 10 seeds in either environment. We believe this is consistent with the role of terminal conditioning. Its main benefit is to mitigate the all-zero candidate scores of MC scoring (Figure 6), and a learned probability at the last hop already addresses this issue.

Implementation note: The only additional component that this variant requires is a second head for the intermediate discount β=0.95\beta = 0.95, because the terminal head is trained at the terminal discount γ=0.99\gamma = 0.99. We trained it using the same architecture, loss and update schedule as our terminal head. We then replaced the sampled reward indicators at the first three hops with the second head's predictions. We ran 10 seeds for 50M transitions, with all other components (GHM, planner, evaluation protocol) unchanged.

Just a small clarification on terminology: we would like to clarify that our terminal head is not a density estimator. It predicts a conditional probability mass that the GHM’s terminal distribution assigns to the goal region.

Q3. Could you clarify how many future states are sampled when running the MC version in the experiments section?

OGPC-MC scores every candidate with N=64N=64 simulations (Table 12, Appendix E.3.3). Each simulation samples one future state per subgoal, so with K=4K=4 subgoals, OGPC-MC draws 64×4=25664 \times 4 = 256 future states per candidate. We note that OGPC-TC uses the same NN as OGPC-MC. We agree that this should be easier to find from the main text. We will state N=64N=64 next to K=4K=4 in the Methods paragraph of Section 4.

(Minor comment) “4.2. Q2. DOES OGPC EXTEND TO HIGH-DIMENSIONAL CONTROL?” Please clarify that this refers to a high-dimensional action space.

Yes, it refers to the action space, but we also note that the state space of the Humanoid tasks is significantly larger as well. The Humanoid tasks have 17 action dimensions, compared to 5 to 8 for the other 6 tasks. Humanoid tasks have state dimensions of 268, compared to 29 in the Ant Maze tasks. The exact details are provided in Table 8 of Appendix E.1. We will specify this in Section 4.2.

References

[1] Bortkiewicz et al. Accelerating Goal-Conditioned RL Algorithms and Research. 2025.

[2] Farebrother et al. Compositional planning with jumpy world models. 2026.

[3] Grathwohl et al. FFJORD: Free-form continuous dynamics for scalable reversible generative models. 2019.

[4] Thakoor et al. Generalised policy improvement with geometric policy composition. 2022.

I thank the authors for their detailed response. While I still maintain that the paper is an incremental contribution over Jumpy World Models (Farebrother et. al., 2026), the method does work and the paper is thorough. I therefore increase my score.

Reviewer_716hOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a principled approach to goal-conditioned RL tasks in which reaching subgoals along a trajectory can help reach the overall goal (both at inference time and during exploration). The key technical insight is based on Geometric Horizon Models (which model the successor measure of a policy) and Geometric Policy Composition (a form of policy sequencing with probabilistic transitions between subpolicies, which enables the successor measure of the sequenced policy to be modeled compositionally). The paper exploits this compositionality to perform planning of which subgoal policies to sequence together. The authors further show that their method benefits from a sampling strategy based on Rao-Blackwellization that lets intermediate success reached along a prefix be useful in reducing variance of the estimate of terminal success. The experimental evaluation is performed in a well-known benchmark suite and the approach shows empirical gains over prior works.

Critical strengths
  1. The paper does a good job of motivating and introducing the mathematical machinery needed to understand the approach: there is adequate signposting and also every technical definition and development is accompanied with relevant citations and a layman’s description. This makes it possible to understand the approach even if the reader is not already familiar with GHMs. Overall, it is a well-written paper.
  2. I appreciate the elegance of the compositional solution, and the theoretical justification for the technique. Papers like this are intellectually stimulating and will inspire researchers to pursue projects with similar “vibes” even in different problem settings.
  3. The approach is empirically justified in a principled experiments section. It shows significant improvements over relevant baselines.
Critical weaknesses

It was somewhat confusing on first reading for me to understand what part of the approach helps with the “sparse” rewards as mentioned in the abstract. I initially thought that the paper would be about some sort of curiosity-driven exploration. But after reading section 3.2 I came to the understanding that this approach does not help the agent stumble upon a goal’s success region for the first time, rather its use of Terminal Rao-Blackwellization makes it so that successes from a rollout prefix are more helpful in estimating the terminal success probability even when actually terminating in a success state is rare. I would say that this is not actually a very critical weakness, because I was able to figure it out when I read the relevant technical section of the paper, but it would be good to make it a bit more obvious.

Questions for authors

How does your approach help with sparse rewards? Is my understanding (that the Terminal Rao-Blackwellization helps with sparse rewards by letting successes encountered in prefixes be more useful in estimating terminal success probability) correct? It would be nice to have some clarification. If I misinterpreted the paper, then please let me know, and also please say what part of the paper you will update to make it more clear.

2 comments

We sincerely thank the reviewer for their positive and thoughtful assessment of the paper. We also appreciate that they took the time to work through the initially unclear connection to sparse rewards. We address your question below, please let us know if there’s anything else you’d like us to clarify.

How does your approach help with sparse rewards? Is my understanding (that the Terminal Rao-Blackwellization helps with sparse rewards by letting successes encountered in prefixes be more useful in estimating terminal success probability) correct?

Your understanding is correct in the sense that TC keeps plan scores informative even when sampled terminal successes are rare. We would like to clarify that the benefit of TC does not specifically rely on successes encountered earlier in the simulated plan before the final subgoal. The terminal probability is conditioned on the state and action from which the final subgoal is pursued, so it can be positive even when there are no successes before that point.

I came to the understanding that this approach does not help the agent stumble upon a goal’s success region for the first time

You are right that TC is a plan-scoring estimator and does not introduce any intrinsic curiosity or novelty bonus. However, TC’s plan can still improve exploration through the subgoal sequences OGPC executes during data collection. For example, in the early training stage of Ant Big Maze, we find that the MC estimator frequently scores all candidates zero and keeps the direct-goal plan, whereas TC’s planner frequently commands intermediate subgoals (Figure 6).

We will revise the abstract to describe OGPC as a method that guides data collection through subgoal planning and clarify TC’s role in sparse-reward plan scoring in Section 3.2, stating that it does not necessarily depend on successes along the prefix.

Reviewer_716hOpenReview ↗

Looks good to me

I would like to thank the authors for their helpful response, and I believe that with the (minor) revisions they propose, the paper will be extremely clear. I maintain my rating (4: Clear Acceptance).

Reviewer_6Bb8OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a method for jointly learning policies and their occupancies in sparse-reward settings. The key idea is to replace evaluating the probability of reaching that state that you used for sampling with the probability that the final sampled state would land int the state of interest---which is the models occupancy mass on the goal set from that point. This is a clever way of reusing an easier likelihood/probability as the "cumulant" here and they draw on the Rao–Blackwell theorem to show that this produces an estimator with the same expected value and the same/lower variance under the learned GHM. They show that this leads to improved performance on a number of robotics tasks.

Critical strengths

In the predictive representation/generalized value function/successor representation literature, this paper tackles a very important problem: jointly learning policies and their occupancies in sparse-reward settings.

Critical weaknesses

The experiments could be more exciting but this is a very nice paper in the relevant literature.

Questions for authors

How do you get the seed direct-goal plan? How do you handle an initially inaccurate model? Or is this not a problem in your experiments? How does your method relate to hindsight-experience replay? It seems like a variant extended to learning geometric horizon models? Why "Time at goal" as a measure instead of some binary "in goal region" metric?

1 comment

We sincerely thank the reviewer for their positive and thoughtful assessment of the paper. We hope to address your questions below. Please let us know if there’s anything else you’d like us to clarify.

Q1. How do you get the direct-goal plan?

Thank you for the question. We define the direct-goal plan as follows:

  • We set all planned subgoals to be equal to the task goal, such that g1=⋯=gK=g⋆g_1=\cdots=g_K=g^\star, and sample its first action from the current goal conditioned policy a0∼πθ(⋅∣s,g⋆)a_0 \sim \pi_\theta(\cdot \mid s, g^\star) (Algorithm 1, line 5). This yields the candidate c0=(a0,g⋆,…,g⋆)c_0 = (a_0, g^\star,\ldots,g^\star) (Algorithm 1, line 6).
  • We score c0c_0 together with other candidate subgoal sequences. This allows the agent to propose a candidate subgoal sequence only if its score is strictly higher than the direct-goal plan.

We emphasize that this direct-goal plan is not derived from additional optimization.

Q2. How do you handle an initially inaccurate model?

You are right that the model (GHM) will be initially inaccurate, but we mitigate this through an initial policy warm-up and receding-horizon planning:

  • Warmup before planning: We first collect experience using the randomly-initialized goal-conditioned policy alone. We then train the policy and GHM after 1000 collected transitions per environment. The agent starts planning after 2000 steps per environment (Appendix E.2). This is consistent with previously-established ideas around training the model before enabling MPC [2], and the delayed use of world model predictions [3].
  • Replanning: We use receding-horizon MPC-style planning, where the agent’s policy follows only the first-subgoal of the selected candidate for a short commitment duration before it replans from the actual observed state (Section 3.1, Appendix E.3.3).

We believe these choices help the agent avoid planning with an entirely untrained model. For qualitative examples of the GHM’s predicted future states, please refer to Appendix G.

Q3. How does your method relate to hindsight-experience replay?

Thank you for pointing out this connection. OGPC uses established hindsight goal relabelling procedures for training the policy and the GHM. We would like to clarify that we are not proposing a new goal relabelling method.

  • Policy Learning: Our policy is trained using the same hindsight-relabelling procedure as our CRL baseline [5] (Appendix E.2 and Table 9).
  • GHM command relabelling: We adapt the hindsight goal-relabelling approach in [4] for GHM training as follows: for each sampled transition (s,a,s′)(s, a, s^\prime), we either keep the executed commanded goal gcmdg^{\mathrm{cmd}} (the task goal or a subgoal selected by the planner) with probability 0.5, or replace it with a goal achieved later in the same episode otherwise (Appendix E.3.1, Table 10).
  • Relationship to HER: OGPC adopts the underlying principle of HER when it comes to reusing the collected experience from the agent under alternative goals. However, whereas HER recomputes the rewards for the replacement goal, we train the GHM to predict future state occupancy of the policy conditioned on the selected goal gg without task-reward or success labels.

We will clarify the relationship to HER in the main text and specify the goal-sampling distribution more explicitly in Appendix E.3.1.

Q4. Why "Time at goal" as a measure instead of some binary "in goal region" metric?

We report “Time at Goal” because it captures both the arrival time and the agent’s subsequent goal occupancy duration. “Time at Goal” counts the number of steps the agent spends in the goal region during a 1000-step episode. A binary success metric measures the outcome of whether the goal region is reached, so it conveys neither the arrival time nor the goal occupancy duration.

For example, in the Humanoid tasks, a humanoid agent might briefly reach the goal before losing balance and leaving the goal region, whereas a more competent agent reaches the goal and remains there for many more steps before the episode terminates. Both episodes count as successful under the binary “Success Rate” metric, but “Time at Goal” allows us to distinguish their behaviors.

We also report the binary metric in the form of Success Rate (i.e. whether the goal region is reached at least once in an episode) as defined in JaxGCRL [1]. We define this metric in Section 4 and report Success Rates across all 9 tasks in Appendix F.1.

References

[1] Bortkiewicz et al. Accelerating Goal-Conditioned RL Algorithms and Research. 2025.

[2] Chua et al. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. 2018.

[3] Dedieu et al. Improving transformer world models for data-efficient RL. 2025.

[4] Farebrother et al. Compositional planning with jumpy world models. 2026.

[5] Wang et al. 1000 layer networks for self-supervised RL: Scaling depth can enable new goal-reaching capabilities. 2025.

Comments on the paper

1 comment

We thank all three reviewers for their time, careful reading, and constructive feedback. Their questions substantially helped us refine the paper and its narrative. We address each point below and will incorporate these improvements into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.